Max level shown:
1 [ Sep 23 18:19:12 Enabled. ]
2 [ Sep 23 18:19:12 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Sep 23 18:19:12 Method "start" exited with status 0. ]
4 [ Sep 23 18:19:12 Rereading configuration. ]
5 [ Sep 23 18:19:12 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-09-23T18:19:12.370ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-09-23T18:19:12.371ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-09-23T18:19:12.423ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-09-23T18:19:12.423ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-09-23T18:19:12.535ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-09-23T18:19:12.545ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-09-23T18:19:12.553ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-09-23T18:19:13.152ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-09-23T18:19:13.290ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-09-23T18:19:13.290ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-09-23T18:19:13.290ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-09-23T18:19:13.290ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-09-23T18:19:13.290ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-09-23T18:19:13.290ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-09-23T18:19:13.293ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-09-23T18:19:13.293ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-09-23T18:19:13.293ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-09-23T18:19:13.293ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-09-23T18:19:13.293ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-09-23T18:19:13.293ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-09-23T18:19:13.293ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-09-23T18:19:13.293ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-09-23T18:19:13.293ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-09-23T18:19:13.305ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-09-23T18:19:13.305ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = f181c9e6-1009-477f-8ae5-05396e960261 (zpool)
322024-09-23T18:19:13.401ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal }
332024-09-23T18:19:13.797ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-09-23T18:19:13.797ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-09-23T18:19:13.797ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-09-23T18:19:13.797ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-09-23T18:19:13.797ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-09-23T18:19:13.797ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-09-23T18:19:13.797ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-09-23T18:19:13.805ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-09-23T18:19:13.805ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool)
422024-09-23T18:19:13.891ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal }
432024-09-23T18:19:14.291ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-09-23T18:19:14.291ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
532024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
542024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
632024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
642024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
652024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
662024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
672024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
712024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
722024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
832024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
842024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
892024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
902024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-09-23T18:19:14.292ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
932024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
942024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
952024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
962024-09-23T18:19:14.292ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
972024-09-23T18:19:14.292ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-09-23T18:19:14.292ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-09-23T18:19:14.292ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-09-23T18:19:14.292ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-09-23T18:19:14.292ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1032024-09-23T18:19:14.292ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-09-23T18:19:14.292ZINFOSledAgent (bootstore): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-09-23T18:19:14.292ZINFOSledAgent (bootstore): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-09-23T18:19:14.292ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-09-23T18:19:14.292ZINFOSledAgent (bootstore): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-09-23T18:19:14.292ZINFOSledAgent (bootstore): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-09-23T18:19:14.292ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-09-23T18:19:14.292ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-09-23T18:19:14.292ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-09-23T18:19:14.303ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:737
1132024-09-23T18:19:14.304ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1078, tv_nsec: 364874874 } time_to_next_cleanup = 599.999917358s
1142024-09-23T18:19:14.304ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1152024-09-23T18:19:14.304ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3917
1162024-09-23T18:19:14.304ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-09-23T18:19:14.304ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-09-23T18:19:14.307ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-09-23T18:19:14.308ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-09-23T18:19:14.309ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-09-23T18:19:14.309ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-09-23T18:19:14.309ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-09-23T18:19:14.309ZINFOSledAgent (RSS): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-09-23T18:19:14.309ZINFOSledAgent (RSS): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-09-23T18:19:14.309ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-09-23T18:19:14.309ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-09-23T18:19:14.309ZINFOSledAgent (RSS): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-09-23T18:19:14.309ZINFOSledAgent (RSS): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-09-23T18:19:14.309ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-09-23T18:19:14.309ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-09-23T18:19:14.309ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-09-23T18:19:14.309ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-09-23T18:19:14.309ZINFOSledAgent (RSS): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-09-23T18:19:14.309ZINFOSledAgent (RSS): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-09-23T18:19:14.310ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 84f7defa-11d2-40bb-a94d-3ddc88dcf69e, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 6061f953-e31a-4a2a-80c5-755d0e0127f8, rack_id: 84f7defa-11d2-40bb-a94d-3ddc88dcf69e, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-09-23T18:19:14.310ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-09-23T18:19:14.310ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-09-23T18:19:14.310ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-09-23T18:19:14.310ZINFOSledAgent (bootstore): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-09-23T18:19:14.310ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-09-23T18:19:14.310ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 6061f953-e31a-4a2a-80c5-755d0e0127f8, rack_id: 84f7defa-11d2-40bb-a94d-3ddc88dcf69e, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-09-23T18:19:14.310ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4214
1432024-09-23T18:19:14.310ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4596
1442024-09-23T18:19:14.315ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-09-23T18:19:14.321ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-09-23T18:19:14.322ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-09-23T18:19:14.326ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-09-23T18:19:14.326ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-09-23T18:19:14.326ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:a02a::1]:63109
1502024-09-23T18:19:14.326ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 6061f953-e31a-4a2a-80c5-755d0e0127f8, rack_id: 84f7defa-11d2-40bb-a94d-3ddc88dcf69e, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:361
1512024-09-23T18:19:14.326ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:376
1522024-09-23T18:19:14.326ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-09-23T18:19:14.326ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1542024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1552024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1562024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1572024-09-23T18:19:14.326ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1582024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-09-23T18:19:14.326ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-09-23T18:19:14.327ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-09-23T18:19:14.327ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-09-23T18:19:14.344ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-09-23T18:19:14.364ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-09-23T18:19:14.382ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-09-23T18:19:14.388ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
1682024-09-23T18:19:14.388ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-09-23T18:19:14.388ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
1702024-09-23T18:19:14.388ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1712024-09-23T18:19:14.392ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 131.305564ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5285db37-6c46-45a1-9883-85634ab43718", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5285db37-6c46-45a1-9883-85634ab43718" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-09-23T18:19:14.401ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_f181c9e6-1009-477f-8ae5-05396e960261/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-09-23T18:19:14.423ZINFOSledAgent: successfully created encrypted zvol: name="oxi_f181c9e6-1009-477f-8ae5-05396e960261/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-09-23T18:19:14.423ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_f181c9e6-1009-477f-8ae5-05396e960261/swap" file = sled-agent/src/swap_device.rs:96
1752024-09-23T18:19:14.435ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
1762024-09-23T18:19:14.435ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:130
1772024-09-23T18:19:14.435ZINFOSledAgent (BackingFs): Ensuring dataset oxi_f181c9e6-1009-477f-8ae5-05396e960261/backing/fmd file = sled-agent/src/backing_fs.rs:140
1782024-09-23T18:19:14.523ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:175
1792024-09-23T18:19:14.525ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1802024-09-23T18:19:14.526ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 176.193618ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "42e056ac-fb6b-4d63-88d3-b78ff8381908", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "42e056ac-fb6b-4d63-88d3-b78ff8381908" } file = clients/ddm-admin-client/src/lib.rs:177
1812024-09-23T18:19:14.703ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-09-23T18:19:14.704ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 162.693886ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "40f7784b-1a2b-4fca-a444-44732538e4d2", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "40f7784b-1a2b-4fca-a444-44732538e4d2" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-09-23T18:19:14.742ZINFOSledAgent (BackingFs): Mounting oxi_f181c9e6-1009-477f-8ae5-05396e960261/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:183
1842024-09-23T18:19:14.755ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:191
1852024-09-23T18:19:14.755ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:191
1862024-09-23T18:19:14.755ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:191
1872024-09-23T18:19:14.755ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:197
1882024-09-23T18:19:14.868ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-09-23T18:19:14.870ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 98.578617ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cd6d8d8b-d019-46aa-a097-2727dd10a378", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cd6d8d8b-d019-46aa-a097-2727dd10a378" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-09-23T18:19:14.969ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-09-23T18:19:14.970ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 187.211261ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "665f801b-0e8f-4a42-8119-3a36cef2862f", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "665f801b-0e8f-4a42-8119-3a36cef2862f" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-09-23T18:19:15.158ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-09-23T18:19:15.159ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 149.793824ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "53dff783-6c89-4efe-8fc3-fea26045a7b7", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "53dff783-6c89-4efe-8fc3-fea26045a7b7" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-09-23T18:19:15.310ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-09-23T18:19:15.310ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 225.310672ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "28df1328-92f8-47b8-9f36-2571636e4750", "content-length": "124", "date": "Mon, 23 Sep 2024 18:19:14 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "28df1328-92f8-47b8-9f36-2571636e4750" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-09-23T18:19:15.475ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
1972024-09-23T18:19:15.480ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:53690 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
1982024-09-23T18:19:15.481ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
1992024-09-23T18:19:15.481ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:53690 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2002024-09-23T18:19:15.482ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 267.226776ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2012024-09-23T18:19:15.483ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2022024-09-23T18:19:15.483ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2032024-09-23T18:19:15.483ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:921 underlay_address = fd00:1122:3344:101::1
2042024-09-23T18:19:15.483ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
2052024-09-23T18:19:15.537ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2062024-09-23T18:19:15.751ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 283.718827ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2072024-09-23T18:19:16.037ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 672.045405ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2082024-09-23T18:19:16.711ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.097113061s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2092024-09-23T18:19:17.812ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.887996607s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2102024-09-23T18:19:18.021ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2670
2112024-09-23T18:19:18.021ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2888
2122024-09-23T18:19:18.021ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2540
2132024-09-23T18:19:18.021ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2587
2142024-09-23T18:19:18.021ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2969
2152024-09-23T18:19:18.021ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3011
2162024-09-23T18:19:18.023ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2813
2172024-09-23T18:19:18.025ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2581
2182024-09-23T18:19:18.025ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='ffa8d3a1-6c79-49f7-8522-b5d1b70dcafa'/> <propval type="astring" name="rack_id" value='84f7defa-11d2-40bb-a94d-3ddc88dcf69e'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe6f:a7bc'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='6061f953-e31a-4a2a-80c5-755d0e0127f8'/> <propval type="astring" name="rack_id" value='84f7defa-11d2-40bb-a94d-3ddc88dcf69e'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='6061f953-e31a-4a2a-80c5-755d0e0127f8'/> <propval type="astring" name="rack_uuid" value='84f7defa-11d2-40bb-a94d-3ddc88dcf69e'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='6061f953-e31a-4a2a-80c5-755d0e0127f8'/> <propval type="astring" name="rack_uuid" value='84f7defa-11d2-40bb-a94d-3ddc88dcf69e'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2192024-09-23T18:19:18.025ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2202024-09-23T18:19:18.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 72.858657ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-09-23T18:19:18.469ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 116.586211ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-09-23T18:19:18.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 215.536862ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-09-23T18:19:19.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 209.097427ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-09-23T18:19:19.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 603.761664ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-09-23T18:19:20.056ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.204164013s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-09-23T18:19:20.703ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.994276486s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2272024-09-23T18:19:21.494ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 675.584552ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2282024-09-23T18:19:22.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.000289597s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2292024-09-23T18:19:23.920ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2302024-09-23T18:19:23.920ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:431
2312024-09-23T18:19:23.920ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2322024-09-23T18:19:23.920ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-09-23T18:19:23.920ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2342024-09-23T18:19:23.920ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2352024-09-23T18:19:23.920ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2362024-09-23T18:19:23.920ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3917
2372024-09-23T18:19:23.920ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:a02a::1]:63109
2382024-09-23T18:19:24.160ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2392024-09-23T18:19:24.160ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2402024-09-23T18:19:24.160ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-09-23T18:19:24.160ZINFOSledAgent (RSS): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-09-23T18:19:24.160ZINFOSledAgent (RSS): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-09-23T18:19:24.160ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-09-23T18:19:24.160ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-09-23T18:19:24.160ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-09-23T18:19:24.165ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4238 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2472024-09-23T18:19:24.165ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4254
2482024-09-23T18:19:24.165ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2492024-09-23T18:19:24.445ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61863
2502024-09-23T18:19:24.445ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2512024-09-23T18:19:24.445ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 327 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61863 req_id = 3a366236-e402-4a3f-9523-49aaa06cea9a response_code = 200 uri = /inventory
2522024-09-23T18:19:24.475ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35795
2532024-09-23T18:19:24.475ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35795 req_id = fef4389c-1496-4bd6-bf0d-614b9a25687b response_code = 200 uri = /sled-role
2542024-09-23T18:19:24.476ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2552024-09-23T18:19:24.476ZINFOSledAgent (RSS): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2562024-09-23T18:19:24.476ZINFOSledAgent (RSS): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2572024-09-23T18:19:24.476ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:1001
2582024-09-23T18:19:24.508ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 3827056b-e406-4a68-93f3-568566d559d1, pool_id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 93839df8-3dcf-4a37-890c-73aed2865ed9, pool_id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: cf16bf06-9255-4e7e-8d56-e636fee25968, pool_id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: f435611f-d792-4b54-b235-20c924e1da04, pool_id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: d2b37a74-7124-46ba-b4f7-fa305f81dc3d, pool_id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: a1f7c11c-51e4-4016-aaf7-293e7b0b917f, pool_id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 5255e03a-994b-42b5-ac41-0ef6fa7109e0, pool_id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 8ed5ba14-f08a-4440-978e-29c3b8f68702, pool_id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 69627f57-c883-41b0-93c8-e3e42788c84f, pool_id: 73f13013-70b2-4020-853f-5a70920c296f (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2592024-09-23T18:19:24.508ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54330
2602024-09-23T18:19:24.508ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2612024-09-23T18:19:24.508ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 3827056b-e406-4a68-93f3-568566d559d1, pool_id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 93839df8-3dcf-4a37-890c-73aed2865ed9, pool_id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: cf16bf06-9255-4e7e-8d56-e636fee25968, pool_id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: f435611f-d792-4b54-b235-20c924e1da04, pool_id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: d2b37a74-7124-46ba-b4f7-fa305f81dc3d, pool_id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: a1f7c11c-51e4-4016-aaf7-293e7b0b917f, pool_id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 5255e03a-994b-42b5-ac41-0ef6fa7109e0, pool_id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 8ed5ba14-f08a-4440-978e-29c3b8f68702, pool_id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 69627f57-c883-41b0-93c8-e3e42788c84f, pool_id: 73f13013-70b2-4020-853f-5a70920c296f (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2622024-09-23T18:19:24.508ZINFOSledAgent (StorageManager): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2632024-09-23T18:19:24.508ZINFOSledAgent (StorageManager): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2642024-09-23T18:19:24.508ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2652024-09-23T18:19:24.508ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2662024-09-23T18:19:24.508ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2672024-09-23T18:19:24.508ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2682024-09-23T18:19:24.508ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2692024-09-23T18:19:24.669ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2702024-09-23T18:19:24.669ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool)
2712024-09-23T18:19:24.922ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2722024-09-23T18:19:25.656ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4080186723256583263 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2732024-09-23T18:19:25.656ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 18396305621007731405 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2742024-09-23T18:19:26.124ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2752024-09-23T18:19:26.125ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
2762024-09-23T18:19:26.125ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }
2772024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:229
2782024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:231
2792024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:235
2802024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:237
2812024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2822024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2832024-09-23T18:19:26.135ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
2842024-09-23T18:19:26.169ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2852024-09-23T18:19:26.178ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_016c0356-f4b4-4d6b-8c24-edfd2951cfe3/crypt/zone file = sled-storage/src/dataset.rs:292
2862024-09-23T18:19:26.350ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_016c0356-f4b4-4d6b-8c24-edfd2951cfe3 file = sled-storage/src/dataset.rs:378
2872024-09-23T18:19:26.358ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2882024-09-23T18:19:26.358ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2892024-09-23T18:19:26.358ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2902024-09-23T18:19:26.358ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2912024-09-23T18:19:26.358ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2922024-09-23T18:19:26.368ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2932024-09-23T18:19:26.368ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool)
2942024-09-23T18:19:26.458ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }
2952024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:229
2962024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:231
2972024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:235
2982024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:237
2992024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3002024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3012024-09-23T18:19:26.466ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3022024-09-23T18:19:26.501ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3032024-09-23T18:19:26.509ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_b5c5f1b9-c752-4261-a49d-06cee1ff1f1a/crypt/zone file = sled-storage/src/dataset.rs:292
3042024-09-23T18:19:26.680ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_b5c5f1b9-c752-4261-a49d-06cee1ff1f1a file = sled-storage/src/dataset.rs:378
3052024-09-23T18:19:26.688ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3062024-09-23T18:19:26.688ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3072024-09-23T18:19:26.688ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3082024-09-23T18:19:26.688ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3092024-09-23T18:19:26.688ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3102024-09-23T18:19:26.698ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3112024-09-23T18:19:26.698ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool)
3122024-09-23T18:19:26.791ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }
3132024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:229
3142024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:231
3152024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:235
3162024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:237
3172024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3182024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3192024-09-23T18:19:26.799ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3202024-09-23T18:19:26.834ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3212024-09-23T18:19:26.843ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e381d5a5-91f1-4b1c-b0ad-128e813a0a94/crypt/zone file = sled-storage/src/dataset.rs:292
3222024-09-23T18:19:27.014ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e381d5a5-91f1-4b1c-b0ad-128e813a0a94 file = sled-storage/src/dataset.rs:378
3232024-09-23T18:19:27.023ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3242024-09-23T18:19:27.023ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3252024-09-23T18:19:27.023ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3262024-09-23T18:19:27.023ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3272024-09-23T18:19:27.023ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3282024-09-23T18:19:27.033ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3292024-09-23T18:19:27.033ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool)
3302024-09-23T18:19:27.126ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }
3312024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:229
3322024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:231
3332024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:235
3342024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:237
3352024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3362024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3372024-09-23T18:19:27.134ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3382024-09-23T18:19:27.139ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
3392024-09-23T18:19:27.171ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3402024-09-23T18:19:27.178ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_67a20504-7be3-48ff-bb32-7f9411b1342b/crypt/zone file = sled-storage/src/dataset.rs:292
3412024-09-23T18:19:27.348ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_67a20504-7be3-48ff-bb32-7f9411b1342b file = sled-storage/src/dataset.rs:378
3422024-09-23T18:19:27.356ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3432024-09-23T18:19:27.356ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3442024-09-23T18:19:27.356ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3452024-09-23T18:19:27.356ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3462024-09-23T18:19:27.356ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3472024-09-23T18:19:27.365ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3482024-09-23T18:19:27.365ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool)
3492024-09-23T18:19:27.457ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }
3502024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:229
3512024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:231
3522024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:235
3532024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:237
3542024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3552024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3562024-09-23T18:19:27.464ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3572024-09-23T18:19:27.499ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3582024-09-23T18:19:27.506ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4edf146e-3ed8-4c23-b131-d4ed33864007/crypt/zone file = sled-storage/src/dataset.rs:292
3592024-09-23T18:19:27.673ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4edf146e-3ed8-4c23-b131-d4ed33864007 file = sled-storage/src/dataset.rs:378
3602024-09-23T18:19:27.681ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3612024-09-23T18:19:27.681ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3622024-09-23T18:19:27.681ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3632024-09-23T18:19:27.681ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3642024-09-23T18:19:27.681ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3652024-09-23T18:19:27.690ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3662024-09-23T18:19:27.690ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool)
3672024-09-23T18:19:27.787ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool), kind: External }
3682024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:229
3692024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:231
3702024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:235
3712024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:237
3722024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3732024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3742024-09-23T18:19:27.797ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3752024-09-23T18:19:27.833ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3762024-09-23T18:19:27.841ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_17d503a6-b44e-42a3-ae74-5b19be755ad5/crypt/zone file = sled-storage/src/dataset.rs:292
3772024-09-23T18:19:28.012ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_17d503a6-b44e-42a3-ae74-5b19be755ad5 file = sled-storage/src/dataset.rs:378
3782024-09-23T18:19:28.020ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3792024-09-23T18:19:28.020ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3802024-09-23T18:19:28.020ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3812024-09-23T18:19:28.020ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3822024-09-23T18:19:28.020ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3832024-09-23T18:19:28.029ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3842024-09-23T18:19:28.029ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool)
3852024-09-23T18:19:28.122ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External }
3862024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:229
3872024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:231
3882024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:235
3892024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:237
3902024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3912024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3922024-09-23T18:19:28.130ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
3932024-09-23T18:19:28.141ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
3942024-09-23T18:19:28.165ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3952024-09-23T18:19:28.173ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1f7ede35-40db-4a1a-8087-fc87b6705700/crypt/zone file = sled-storage/src/dataset.rs:292
3962024-09-23T18:19:28.360ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1f7ede35-40db-4a1a-8087-fc87b6705700 file = sled-storage/src/dataset.rs:378
3972024-09-23T18:19:28.369ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3982024-09-23T18:19:28.369ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3992024-09-23T18:19:28.369ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4002024-09-23T18:19:28.369ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4012024-09-23T18:19:28.369ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4022024-09-23T18:19:28.378ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
4032024-09-23T18:19:28.378ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool)
4042024-09-23T18:19:28.474ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External }
4052024-09-23T18:19:28.482ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:229
4062024-09-23T18:19:28.483ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:231
4072024-09-23T18:19:28.483ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:235
4082024-09-23T18:19:28.483ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:237
4092024-09-23T18:19:28.483ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4102024-09-23T18:19:28.483ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4112024-09-23T18:19:28.483ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
4122024-09-23T18:19:28.520ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4132024-09-23T18:19:28.528ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c6321c72-f89d-471a-8986-1f1bf65a3de2/crypt/zone file = sled-storage/src/dataset.rs:292
4142024-09-23T18:19:28.705ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c6321c72-f89d-471a-8986-1f1bf65a3de2 file = sled-storage/src/dataset.rs:378
4152024-09-23T18:19:28.713ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4162024-09-23T18:19:28.713ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4172024-09-23T18:19:28.713ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4182024-09-23T18:19:28.713ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4192024-09-23T18:19:28.713ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4202024-09-23T18:19:28.722ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4212024-09-23T18:19:28.722ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 73f13013-70b2-4020-853f-5a70920c296f (zpool)
4222024-09-23T18:19:28.819ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:185 zpool = ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External }
4232024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:229
4242024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:231
4252024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:235
4262024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:237
4272024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4282024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4292024-09-23T18:19:28.826ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:249
4302024-09-23T18:19:28.861ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4312024-09-23T18:19:28.868ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_73f13013-70b2-4020-853f-5a70920c296f/crypt/zone file = sled-storage/src/dataset.rs:292
4322024-09-23T18:19:29.042ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_73f13013-70b2-4020-853f-5a70920c296f file = sled-storage/src/dataset.rs:378
4332024-09-23T18:19:29.051ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4342024-09-23T18:19:29.051ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4352024-09-23T18:19:29.051ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4362024-09-23T18:19:29.051ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f181c9e6-1009-477f-8ae5-05396e960261 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: a6d75f0e-db4f-4052-94fb-1f0675a27263 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4372024-09-23T18:19:29.051ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4382024-09-23T18:19:29.051ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4392024-09-23T18:19:29.051ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4402024-09-23T18:19:29.051ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4412024-09-23T18:19:29.051ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4422024-09-23T18:19:29.051ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4432024-09-23T18:19:29.051ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4442024-09-23T18:19:29.051ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4452024-09-23T18:19:29.051ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4462024-09-23T18:19:29.051ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4472024-09-23T18:19:29.051ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:773 gen = Generation(1)
4482024-09-23T18:19:29.051ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4492024-09-23T18:19:29.051ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4543276 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:54330 req_id = fbde5aa4-40a8-450c-8f08-76ef9537b088 response_code = 200 uri = /omicron-physical-disks
4502024-09-23T18:19:29.082ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: df62fab1-4575-460a-9193-f90470451eed (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a816df87-b330-4b29-8c67-45ce2059ce69 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4512024-09-23T18:19:29.082ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64086
4522024-09-23T18:19:29.083ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4532024-09-23T18:19:29.083ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4542024-09-23T18:19:29.134ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4552024-09-23T18:19:29.135ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }
4562024-09-23T18:19:29.146ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4572024-09-23T18:19:29.186ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4582024-09-23T18:19:29.186ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4592024-09-23T18:19:29.237ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4602024-09-23T18:19:29.237ZINFOSledAgent (ServiceManager): No ledger in /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-zones.json file = common/src/ledger.rs:188
4612024-09-23T18:19:29.237ZINFOSledAgent (ServiceManager): No ledger in /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-zones.json file = common/src/ledger.rs:188
4622024-09-23T18:19:29.237ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4632024-09-23T18:19:29.280ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4642024-09-23T18:19:29.280ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-09-23T18:19:29.280ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4662024-09-23T18:19:29.332ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1 file = illumos-utils/src/zone.rs:316
4672024-09-23T18:19:29.335ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed file = illumos-utils/src/zone.rs:316
4682024-09-23T18:19:29.339ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69 file = illumos-utils/src/zone.rs:316
4692024-09-23T18:19:29.352ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1 file = illumos-utils/src/zone.rs:354
4702024-09-23T18:19:29.356ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed file = illumos-utils/src/zone.rs:354
4712024-09-23T18:19:29.360ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69 file = illumos-utils/src/zone.rs:354
4722024-09-23T18:19:30.148ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
4732024-09-23T18:19:30.845ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4742024-09-23T18:19:30.845ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4752024-09-23T18:19:30.845ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4762024-09-23T18:19:31.327ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4772024-09-23T18:19:31.806ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4782024-09-23T18:19:31.807ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-09-23T18:19:31.807ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4802024-09-23T18:19:31.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 25.953147ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4812024-09-23T18:19:31.816ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4822024-09-23T18:19:31.820ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
4832024-09-23T18:19:31.851ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 134.531356ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4842024-09-23T18:19:31.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 47.591831ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4852024-09-23T18:19:31.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 28.521992ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
4862024-09-23T18:19:31.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 132.617559ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
4872024-09-23T18:19:31.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 149.886062ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4882024-09-23T18:19:32.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 62.085865ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4892024-09-23T18:19:32.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 215.806169ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4902024-09-23T18:19:32.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 172.297057ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
4912024-09-23T18:19:32.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 342.517975ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4922024-09-23T18:19:32.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 264.15794ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
4932024-09-23T18:19:32.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 595.964029ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4942024-09-23T18:19:32.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 581.705685ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4952024-09-23T18:19:32.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 1.160574002s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
4962024-09-23T18:19:32.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 779.167207ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4972024-09-23T18:19:33.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 924.312194ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
4982024-09-23T18:19:33.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 900.363102ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
4992024-09-23T18:19:33.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 1.09664447s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
5002024-09-23T18:19:33.922ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10707974267007014508 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
5012024-09-23T18:19:33.922ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14855376552835773408 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
5022024-09-23T18:19:34.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 793.729189ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
5032024-09-23T18:19:34.310ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4262
5042024-09-23T18:19:34.310ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4272 gateway = fd00:1122:3344:101::1
5052024-09-23T18:19:34.316ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4351
5062024-09-23T18:19:34.445ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4399
5072024-09-23T18:19:34.446ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4426
5082024-09-23T18:19:34.485ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4439
5092024-09-23T18:19:34.485ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4303
5102024-09-23T18:19:34.538ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4345
5112024-09-23T18:19:34.538ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4406 rack_subnet = fd00:1122:3344::
5122024-09-23T18:19:34.557ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4417
5132024-09-23T18:19:34.557ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4459
5142024-09-23T18:19:34.644ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4491
5152024-09-23T18:19:34.644ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4497
5162024-09-23T18:19:34.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 1.290407822s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
5172024-09-23T18:19:34.754ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4538
5182024-09-23T18:19:34.754ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5192024-09-23T18:19:34.787ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/6061f953-e31a-4a2a-80c5-755d0e0127f8): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
5202024-09-23T18:19:34.787ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 74.357716ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5212024-09-23T18:19:34.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 991.43428ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
5222024-09-23T18:19:34.863ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 92.396649ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5232024-09-23T18:19:34.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 1.114934499s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
5242024-09-23T18:19:34.958ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5252024-09-23T18:19:34.990ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5262024-09-23T18:19:34.990ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5272024-09-23T18:19:34.991ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5282024-09-23T18:19:34.991ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5292024-09-23T18:19:35.028ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
5302024-09-23T18:19:35.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:51900
5312024-09-23T18:19:35.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:51900 req_id = 5e77d1a9-e3a8-4931-be48-c14f22a0f9e5 response_code = 200 uri = /sled-identifiers
5322024-09-23T18:19:35.078ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5332024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5342024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5352024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5362024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5372024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5382024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5392024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5402024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5412024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5422024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5432024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5442024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5452024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5462024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5472024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5482024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5492024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5502024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5512024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5522024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5532024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5542024-09-23T18:19:35.078ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5552024-09-23T18:19:35.659ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4080186723256583263 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
5562024-09-23T18:19:35.659ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 18396305621007731405 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
5572024-09-23T18:19:35.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1") failed: Property not found. retry in 882.110717ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
5582024-09-23T18:19:35.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed") failed: Property not found. retry in 854.720913ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
5592024-09-23T18:19:35.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69") failed: Property not found. retry in 1.338905308s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
5602024-09-23T18:19:37.355ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_8a315cf1-5df7-4f3d-9fde-62f2891368e1
5612024-09-23T18:19:37.355ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_df62fab1-4575-460a-9193-f90470451eed
5622024-09-23T18:19:37.355ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_internal_dns_a816df87-b330-4b29-8c67-45ce2059ce69
5632024-09-23T18:19:37.355ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5642024-09-23T18:19:37.355ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5652024-09-23T18:19:37.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8273161 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:64086 req_id = 15b7cc35-a84b-4e7d-91b8-778f945404de response_code = 204 uri = /omicron-zones
5662024-09-23T18:19:37.358ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5672024-09-23T18:19:37.409ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5682024-09-23T18:19:37.454ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5692024-09-23T18:19:37.500ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5702024-09-23T18:19:37.500ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5712024-09-23T18:19:37.538ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5722024-09-23T18:19:37.539ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5732024-09-23T18:19:37.539ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5742024-09-23T18:19:37.568ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: df62fab1-4575-460a-9193-f90470451eed (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a816df87-b330-4b29-8c67-45ce2059ce69 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: e99136cf-dafe-4a2b-8ac8-d8b3dd8e5585, kind: Service { id: dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e }, name: Name("ntp-dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 80])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5752024-09-23T18:19:37.568ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45437
5762024-09-23T18:19:37.568ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5772024-09-23T18:19:37.568ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5782024-09-23T18:19:37.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5792024-09-23T18:19:37.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }
5802024-09-23T18:19:37.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5812024-09-23T18:19:37.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5822024-09-23T18:19:37.620ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5832024-09-23T18:19:37.621ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-zones.json file = common/src/ledger.rs:177
5842024-09-23T18:19:37.621ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-zones.json file = common/src/ledger.rs:177
5852024-09-23T18:19:37.621ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5862024-09-23T18:19:37.632ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5872024-09-23T18:19:37.664ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5882024-09-23T18:19:37.666ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5892024-09-23T18:19:37.666ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5902024-09-23T18:19:37.695ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 179, 80]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5912024-09-23T18:19:37.695ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = boundary_ntp
5922024-09-23T18:19:37.696ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5932024-09-23T18:19:37.696ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5942024-09-23T18:19:37.696ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
5952024-09-23T18:19:37.698ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5962024-09-23T18:19:37.721ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e file = illumos-utils/src/zone.rs:316
5972024-09-23T18:19:37.739ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e file = illumos-utils/src/zone.rs:354
5982024-09-23T18:19:38.359ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5992024-09-23T18:19:38.359ZINFOSledAgent (ServiceManager): Booting oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6002024-09-23T18:19:38.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 47.27319ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6012024-09-23T18:19:38.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 95.808008ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6022024-09-23T18:19:38.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 260.285409ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6032024-09-23T18:19:38.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 589.446406ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6042024-09-23T18:19:39.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 571.558334ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6052024-09-23T18:19:40.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 559.159519ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6062024-09-23T18:19:40.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 1.410424443s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6072024-09-23T18:19:42.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 928.952347ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6082024-09-23T18:19:43.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e") failed: Property not found. retry in 1.371029587s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6092024-09-23T18:19:43.924ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10707974267007014508 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6102024-09-23T18:19:43.924ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14855376552835773408 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6112024-09-23T18:19:44.465ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e
6122024-09-23T18:19:44.465ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6132024-09-23T18:19:44.465ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6142024-09-23T18:19:44.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6897577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:45437 req_id = 4d75c72f-1ca0-4076-b2e9-c7f5e58f2805 response_code = 204 uri = /omicron-zones
6152024-09-23T18:19:44.468ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6162024-09-23T18:19:44.497ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6172024-09-23T18:19:44.497ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48851
6182024-09-23T18:19:44.506ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6192024-09-23T18:19:44.506ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8471 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48851 req_id = 19ccf7ae-aa76-4d0a-a1fa-3fbbf61e7322 response_code = 500 uri = /timesync
6202024-09-23T18:19:44.507ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6212024-09-23T18:19:44.661ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6222024-09-23T18:19:44.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52197
6232024-09-23T18:19:45.662ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4080186723256583263 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6242024-09-23T18:19:45.662ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 18396305621007731405 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6252024-09-23T18:19:46.753ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15356014710323229544 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6262024-09-23T18:19:46.836ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16187146033903407450 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6272024-09-23T18:19:47.358ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6643521884715850707 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
6282024-09-23T18:19:48.670ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6292024-09-23T18:19:48.670ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 4008495 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52197 req_id = 62c6e306-1a96-4ebe-915b-45a2625e6adb response_code = 500 uri = /timesync
6302024-09-23T18:19:48.672ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6312024-09-23T18:19:48.769ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6322024-09-23T18:19:48.769ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45764
6332024-09-23T18:19:48.778ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6342024-09-23T18:19:48.778ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8526 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45764 req_id = a0fb3af9-622a-4ddf-8eea-2cbac49d09b5 response_code = 500 uri = /timesync
6352024-09-23T18:19:48.781ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6362024-09-23T18:19:49.019ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6372024-09-23T18:19:49.019ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35103
6382024-09-23T18:19:49.028ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3870
6392024-09-23T18:19:49.028ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8447 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35103 req_id = 602c31f4-c460-48e9-8e1e-04acd0e8c2f2 response_code = 500 uri = /timesync
6402024-09-23T18:19:49.030ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6412024-09-23T18:19:49.272ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6422024-09-23T18:19:49.272ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:32768
6432024-09-23T18:19:50.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1007374 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:32768 req_id = 04ddffcb-22b8-4497-b557-eba6017d92fe response_code = 200 uri = /timesync
6442024-09-23T18:19:50.283ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6452024-09-23T18:19:50.283ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6462024-09-23T18:19:50.568ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6472024-09-23T18:19:50.569ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36229
6482024-09-23T18:19:50.578ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8820 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36229 req_id = 1a134633-0768-4eb7-89bd-5d682a836937 response_code = 200 uri = /timesync
6492024-09-23T18:19:50.579ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6502024-09-23T18:19:50.579ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6512024-09-23T18:19:50.940ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6522024-09-23T18:19:50.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42222
6532024-09-23T18:19:50.949ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9029 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42222 req_id = e8e0699f-9b84-49b7-86d0-ddc4582b93e1 response_code = 200 uri = /timesync
6542024-09-23T18:19:50.951ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6552024-09-23T18:19:50.951ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6562024-09-23T18:19:51.138ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6572024-09-23T18:19:51.138ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55155
6582024-09-23T18:19:51.146ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7923 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55155 req_id = 8ec72505-7bf6-4941-a2e9-68c5b723bd93 response_code = 200 uri = /timesync
6592024-09-23T18:19:51.148ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6602024-09-23T18:19:51.148ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6612024-09-23T18:19:52.047ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6622024-09-23T18:19:52.048ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34412
6632024-09-23T18:19:52.068ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 20854 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34412 req_id = 933ed590-efd8-4c8e-9167-d1182f95e10a response_code = 200 uri = /timesync
6642024-09-23T18:19:52.071ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 3625914380, ip_addr: 216.31.16.12, stratum: 3, ref_time: 1727115590.853292, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6652024-09-23T18:19:52.071ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
6662024-09-23T18:19:52.101ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: df62fab1-4575-460a-9193-f90470451eed (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a816df87-b330-4b29-8c67-45ce2059ce69 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: e99136cf-dafe-4a2b-8ac8-d8b3dd8e5585, kind: Service { id: dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e }, name: Name("ntp-dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 80])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3f116413-4e92-4582-b3a5-e25e835c42da (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } } } }, OmicronZoneConfig { id: ffb68cf7-819b-4482-96ea-a92802127e4d (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } } } }, OmicronZoneConfig { id: 65e6efb1-be8e-412e-b4dc-184f63bb1b9e (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } } } }, OmicronZoneConfig { id: 02a2a6f5-4966-4fbe-96a2-cc8c17266bdf (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External } } } }, OmicronZoneConfig { id: 349f8cac-ee32-4b29-afb5-5e8a126d2e81 (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6672024-09-23T18:19:52.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35402
6682024-09-23T18:19:52.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6692024-09-23T18:19:52.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6702024-09-23T18:19:52.122ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6712024-09-23T18:19:52.122ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }
6722024-09-23T18:19:52.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6732024-09-23T18:19:52.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6742024-09-23T18:19:52.159ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f116413-4e92-4582-b3a5-e25e835c42da, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6752024-09-23T18:19:52.159ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3f116413-4e92-4582-b3a5-e25e835c42da, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6762024-09-23T18:19:52.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ffb68cf7-819b-4482-96ea-a92802127e4d, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6772024-09-23T18:19:52.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ffb68cf7-819b-4482-96ea-a92802127e4d, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: Cockroach }, responder: ... }
6782024-09-23T18:19:52.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 65e6efb1-be8e-412e-b4dc-184f63bb1b9e, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6792024-09-23T18:19:52.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 65e6efb1-be8e-412e-b4dc-184f63bb1b9e, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6802024-09-23T18:19:52.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 02a2a6f5-4966-4fbe-96a2-cc8c17266bdf, dataset_name: DatasetName { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6812024-09-23T18:19:52.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 02a2a6f5-4966-4fbe-96a2-cc8c17266bdf, dataset_name: DatasetName { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }, kind: Cockroach }, responder: ... }
6822024-09-23T18:19:52.132ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 349f8cac-ee32-4b29-afb5-5e8a126d2e81, dataset_name: DatasetName { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6832024-09-23T18:19:52.132ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 349f8cac-ee32-4b29-afb5-5e8a126d2e81, dataset_name: DatasetName { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6842024-09-23T18:19:52.186ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6852024-09-23T18:19:52.186ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-zones.json file = common/src/ledger.rs:177
6862024-09-23T18:19:52.186ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-zones.json file = common/src/ledger.rs:177
6872024-09-23T18:19:52.186ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6882024-09-23T18:19:52.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6892024-09-23T18:19:52.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6902024-09-23T18:19:52.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6912024-09-23T18:19:52.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6922024-09-23T18:19:52.270ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6932024-09-23T18:19:52.360ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf file = illumos-utils/src/zone.rs:316
6942024-09-23T18:19:52.364ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da file = illumos-utils/src/zone.rs:316
6952024-09-23T18:19:52.368ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e file = illumos-utils/src/zone.rs:316
6962024-09-23T18:19:52.373ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81 file = illumos-utils/src/zone.rs:316
6972024-09-23T18:19:52.377ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d file = illumos-utils/src/zone.rs:316
6982024-09-23T18:19:52.381ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf file = illumos-utils/src/zone.rs:354
6992024-09-23T18:19:52.386ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da file = illumos-utils/src/zone.rs:354
7002024-09-23T18:19:52.390ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e file = illumos-utils/src/zone.rs:354
7012024-09-23T18:19:52.395ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81 file = illumos-utils/src/zone.rs:354
7022024-09-23T18:19:52.401ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d file = illumos-utils/src/zone.rs:354
7032024-09-23T18:19:55.377ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='02a2a6f5-4966-4fbe-96a2-cc8c17266bdf'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7042024-09-23T18:19:55.377ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7052024-09-23T18:19:55.382ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='3f116413-4e92-4582-b3a5-e25e835c42da'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7062024-09-23T18:19:55.383ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7072024-09-23T18:19:55.469ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='349f8cac-ee32-4b29-afb5-5e8a126d2e81'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7082024-09-23T18:19:55.469ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7092024-09-23T18:19:55.485ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='ffb68cf7-819b-4482-96ea-a92802127e4d'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7102024-09-23T18:19:55.485ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7112024-09-23T18:19:55.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 55.804367ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7122024-09-23T18:19:55.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 37.684829ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7132024-09-23T18:19:55.529ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='65e6efb1-be8e-412e-b4dc-184f63bb1b9e'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7142024-09-23T18:19:55.530ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7152024-09-23T18:19:55.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 68.541227ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7162024-09-23T18:19:55.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 125.209616ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7172024-09-23T18:19:55.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 215.434005ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7182024-09-23T18:19:55.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 34.188646ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7192024-09-23T18:19:56.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 157.119653ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7202024-09-23T18:19:56.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 41.226362ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7212024-09-23T18:19:56.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 290.17983ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7222024-09-23T18:19:57.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 232.777155ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7232024-09-23T18:19:57.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 53.940637ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7242024-09-23T18:19:57.284ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 188.448476ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
7252024-09-23T18:19:57.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 121.274307ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7262024-09-23T18:19:57.573ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 124.421441ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7272024-09-23T18:19:57.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 1.171018781s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7282024-09-23T18:19:57.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 1.003815943s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7292024-09-23T18:19:58.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 52.827285ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7302024-09-23T18:19:58.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 218.104972ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7312024-09-23T18:19:58.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 159.524897ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7322024-09-23T18:19:58.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 238.930974ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7332024-09-23T18:19:58.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 346.085103ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7342024-09-23T18:19:58.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 398.036752ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7352024-09-23T18:19:58.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 208.477776ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7362024-09-23T18:19:58.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 430.563713ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7372024-09-23T18:19:58.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 733.728539ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7382024-09-23T18:19:58.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 605.932502ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7392024-09-23T18:19:58.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.390379336s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7402024-09-23T18:19:59.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 1.358792403s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7412024-09-23T18:19:59.131ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 1.150592269s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7422024-09-23T18:19:59.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 1.002113593s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7432024-09-23T18:19:59.324ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 866.631331ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7442024-09-23T18:20:00.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 1.212343147s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7452024-09-23T18:20:00.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 738.523505ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7462024-09-23T18:20:00.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 865.646995ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7472024-09-23T18:20:00.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 984.922907ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7482024-09-23T18:20:00.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 796.126566ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7492024-09-23T18:20:01.038ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 1.001739813s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7502024-09-23T18:20:01.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 1.332915966s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7512024-09-23T18:20:01.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 625.001708ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7522024-09-23T18:20:01.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 863.305405ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7532024-09-23T18:20:01.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 1.25331135s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7542024-09-23T18:20:01.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 737.149534ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7552024-09-23T18:20:02.048ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 1.192994211s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7562024-09-23T18:20:02.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 1.37920343s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7572024-09-23T18:20:02.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 894.248517ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7582024-09-23T18:20:02.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.206319221s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7592024-09-23T18:20:02.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 656.666487ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7602024-09-23T18:20:03.247ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 1.082544507s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7612024-09-23T18:20:03.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 537.025555ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7622024-09-23T18:20:03.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 1.252406635s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7632024-09-23T18:20:03.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 1.436505359s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7642024-09-23T18:20:03.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.45216189s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7652024-09-23T18:20:03.885ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 1.245087764s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7662024-09-23T18:20:04.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 548.531594ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7672024-09-23T18:20:04.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81") failed: Property not found. retry in 1.058618295s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7682024-09-23T18:20:04.826ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34876
7692024-09-23T18:20:04.826ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34876 req_id = beaea848-3488-4e2d-b8fb-e40bcbc7fce4 response_code = 200 uri = /baseboard
7702024-09-23T18:20:04.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 847.109923ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7712024-09-23T18:20:05.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 892.717605ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7722024-09-23T18:20:05.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d") failed: Property not found. retry in 1.138630135s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7732024-09-23T18:20:05.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.284456895s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7742024-09-23T18:20:05.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf") failed: Property not found. retry in 1.480982287s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7752024-09-23T18:20:05.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da") failed: Property not found. retry in 690.118113ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7762024-09-23T18:20:06.534ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 646.8081ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7772024-09-23T18:20:07.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 728.933743ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7782024-09-23T18:20:07.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 753.862513ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7792024-09-23T18:20:08.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 620.130327ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7802024-09-23T18:20:09.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 741.418097ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7812024-09-23T18:20:10.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.148429934s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7822024-09-23T18:20:11.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.101660779s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7832024-09-23T18:20:12.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.212803147s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7842024-09-23T18:20:13.019ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 352.042325ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
7852024-09-23T18:20:13.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 740.652033ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7862024-09-23T18:20:14.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 687.143153ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7872024-09-23T18:20:15.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e") failed: Property not found. retry in 1.492654539s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7882024-09-23T18:20:16.893ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_02a2a6f5-4966-4fbe-96a2-cc8c17266bdf
7892024-09-23T18:20:16.893ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_3f116413-4e92-4582-b3a5-e25e835c42da
7902024-09-23T18:20:16.893ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_65e6efb1-be8e-412e-b4dc-184f63bb1b9e
7912024-09-23T18:20:16.893ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_349f8cac-ee32-4b29-afb5-5e8a126d2e81
7922024-09-23T18:20:16.893ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_cockroachdb_ffb68cf7-819b-4482-96ea-a92802127e4d
7932024-09-23T18:20:16.893ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7942024-09-23T18:20:16.923ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
7952024-09-23T18:20:16.947ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25162872 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:35402 req_id = 799fdac9-c721-4776-ac83-31a98eb15948 response_code = 204 uri = /omicron-zones
7962024-09-23T18:20:16.979ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37215
7972024-09-23T18:20:17.106ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::6]:32221 file = sled-agent/src/services.rs:3713
7982024-09-23T18:20:23.858ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3731
7992024-09-23T18:20:41.989ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3754
8002024-09-23T18:20:41.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25032431 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37215 req_id = c0d79387-a80e-49e2-a6e9-bd762a6fe577 response_code = 204 uri = /cockroachdb
8012024-09-23T18:20:41.989ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53657
8022024-09-23T18:20:41.990ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 135 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53657 req_id = 6810097a-e37f-4e12-8afa-82454dcb391d response_code = 200 uri = /baseboard
8032024-09-23T18:20:41.995ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 625.338584ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
8042024-09-23T18:20:42.024ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: df62fab1-4575-460a-9193-f90470451eed (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: a816df87-b330-4b29-8c67-45ce2059ce69 (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: e99136cf-dafe-4a2b-8ac8-d8b3dd8e5585, kind: Service { id: dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e }, name: Name("ntp-dc6ddc3e-c0dc-4d16-bf6e-b2db4c94de9e"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 179, 80])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 3f116413-4e92-4582-b3a5-e25e835c42da (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } } } }, OmicronZoneConfig { id: ffb68cf7-819b-4482-96ea-a92802127e4d (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } } } }, OmicronZoneConfig { id: 65e6efb1-be8e-412e-b4dc-184f63bb1b9e (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } } } }, OmicronZoneConfig { id: 02a2a6f5-4966-4fbe-96a2-cc8c17266bdf (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External } } } }, OmicronZoneConfig { id: 349f8cac-ee32-4b29-afb5-5e8a126d2e81 (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External } } } }, OmicronZoneConfig { id: 544f6393-b5b8-4964-b9b8-d8467761342e (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: b9740b46-002a-4447-8a9d-58caea4925b7 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 43821df6-7a7b-49df-9a53-ebf281e00a55 (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: f141ec5c-de3f-4dea-aa70-9c7e0ad6beee (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 336574a9-87bb-450e-9ef7-3aa3e0e79905 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 4c6fe4b9-aa70-4c11-a8cb-78727c3b2349 (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 6837578d-2f18-4d5d-82e1-b69d1cbf6f4c (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 46c88b0c-8be0-41ef-83a1-5caeaba47dc8 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External } } } }, OmicronZoneConfig { id: b473542b-f56f-44f7-ac97-717a7c8f5b46 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External } } } }, OmicronZoneConfig { id: 5d81eec0-e923-40e4-875c-abea1d4aed53 (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External } } } }, OmicronZoneConfig { id: 96a2bbcf-5e5b-4bf1-82f3-5b2051b02712 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External } } } }, OmicronZoneConfig { id: d28c60ef-c4db-4c3e-8e22-baaa61f68157 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External } } } }, OmicronZoneConfig { id: 517b3055-a59f-450e-a349-fc1fc2615512 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool), kind: External } } } }, OmicronZoneConfig { id: 44aeefad-3a65-438e-8790-1c7a579bbb49 (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7c0b2a6b-4c9a-4820-a10b-e91676739409 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2818b80d-465e-48ce-92dc-2436b4b704cd (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8052024-09-23T18:20:42.024ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51030
8062024-09-23T18:20:42.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8072024-09-23T18:20:42.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 8a315cf1-5df7-4f3d-9fde-62f2891368e1, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8082024-09-23T18:20:42.048ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8092024-09-23T18:20:42.048ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: df62fab1-4575-460a-9193-f90470451eed, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: InternalDns }, responder: ... }
8102024-09-23T18:20:42.071ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8112024-09-23T18:20:42.071ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: a816df87-b330-4b29-8c67-45ce2059ce69, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8122024-09-23T18:20:42.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3f116413-4e92-4582-b3a5-e25e835c42da, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8132024-09-23T18:20:42.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3f116413-4e92-4582-b3a5-e25e835c42da, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8142024-09-23T18:20:42.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ffb68cf7-819b-4482-96ea-a92802127e4d, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8152024-09-23T18:20:42.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ffb68cf7-819b-4482-96ea-a92802127e4d, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: Cockroach }, responder: ... }
8162024-09-23T18:20:42.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 65e6efb1-be8e-412e-b4dc-184f63bb1b9e, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8172024-09-23T18:20:42.129ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 65e6efb1-be8e-412e-b4dc-184f63bb1b9e, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8182024-09-23T18:20:42.147ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 02a2a6f5-4966-4fbe-96a2-cc8c17266bdf, dataset_name: DatasetName { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8192024-09-23T18:20:42.147ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 02a2a6f5-4966-4fbe-96a2-cc8c17266bdf, dataset_name: DatasetName { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }, kind: Cockroach }, responder: ... }
8202024-09-23T18:20:42.168ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 349f8cac-ee32-4b29-afb5-5e8a126d2e81, dataset_name: DatasetName { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8212024-09-23T18:20:42.168ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 349f8cac-ee32-4b29-afb5-5e8a126d2e81, dataset_name: DatasetName { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8222024-09-23T18:20:42.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 544f6393-b5b8-4964-b9b8-d8467761342e, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8232024-09-23T18:20:42.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 544f6393-b5b8-4964-b9b8-d8467761342e, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8242024-09-23T18:20:42.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8252024-09-23T18:20:42.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8262024-09-23T18:20:42.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f141ec5c-de3f-4dea-aa70-9c7e0ad6beee, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
8272024-09-23T18:20:42.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f141ec5c-de3f-4dea-aa70-9c7e0ad6beee, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8282024-09-23T18:20:42.430ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 46c88b0c-8be0-41ef-83a1-5caeaba47dc8, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8292024-09-23T18:20:42.430ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 46c88b0c-8be0-41ef-83a1-5caeaba47dc8, dataset_name: DatasetName { pool_name: ZpoolName { id: 016c0356-f4b4-4d6b-8c24-edfd2951cfe3 (zpool), kind: External }, kind: Crucible }, responder: ... }
8302024-09-23T18:20:42.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b473542b-f56f-44f7-ac97-717a7c8f5b46, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8312024-09-23T18:20:42.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: b473542b-f56f-44f7-ac97-717a7c8f5b46, dataset_name: DatasetName { pool_name: ZpoolName { id: b5c5f1b9-c752-4261-a49d-06cee1ff1f1a (zpool), kind: External }, kind: Crucible }, responder: ... }
8322024-09-23T18:20:42.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5d81eec0-e923-40e4-875c-abea1d4aed53, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8332024-09-23T18:20:42.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5d81eec0-e923-40e4-875c-abea1d4aed53, dataset_name: DatasetName { pool_name: ZpoolName { id: e381d5a5-91f1-4b1c-b0ad-128e813a0a94 (zpool), kind: External }, kind: Crucible }, responder: ... }
8342024-09-23T18:20:42.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 96a2bbcf-5e5b-4bf1-82f3-5b2051b02712, dataset_name: DatasetName { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8352024-09-23T18:20:42.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 96a2bbcf-5e5b-4bf1-82f3-5b2051b02712, dataset_name: DatasetName { pool_name: ZpoolName { id: 67a20504-7be3-48ff-bb32-7f9411b1342b (zpool), kind: External }, kind: Crucible }, responder: ... }
8362024-09-23T18:20:42.745ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d28c60ef-c4db-4c3e-8e22-baaa61f68157, dataset_name: DatasetName { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8372024-09-23T18:20:42.745ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: d28c60ef-c4db-4c3e-8e22-baaa61f68157, dataset_name: DatasetName { pool_name: ZpoolName { id: 4edf146e-3ed8-4c23-b131-d4ed33864007 (zpool), kind: External }, kind: Crucible }, responder: ... }
8382024-09-23T18:20:42.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 517b3055-a59f-450e-a349-fc1fc2615512, dataset_name: DatasetName { pool_name: ZpoolName { id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8392024-09-23T18:20:42.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 517b3055-a59f-450e-a349-fc1fc2615512, dataset_name: DatasetName { pool_name: ZpoolName { id: 17d503a6-b44e-42a3-ae74-5b19be755ad5 (zpool), kind: External }, kind: Crucible }, responder: ... }
8402024-09-23T18:20:42.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 44aeefad-3a65-438e-8790-1c7a579bbb49, dataset_name: DatasetName { pool_name: ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8412024-09-23T18:20:42.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 44aeefad-3a65-438e-8790-1c7a579bbb49, dataset_name: DatasetName { pool_name: ZpoolName { id: 1f7ede35-40db-4a1a-8087-fc87b6705700 (zpool), kind: External }, kind: Crucible }, responder: ... }
8422024-09-23T18:20:42.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7c0b2a6b-4c9a-4820-a10b-e91676739409, dataset_name: DatasetName { pool_name: ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8432024-09-23T18:20:42.975ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 7c0b2a6b-4c9a-4820-a10b-e91676739409, dataset_name: DatasetName { pool_name: ZpoolName { id: c6321c72-f89d-471a-8986-1f1bf65a3de2 (zpool), kind: External }, kind: Crucible }, responder: ... }
8442024-09-23T18:20:43.043ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2818b80d-465e-48ce-92dc-2436b4b704cd, dataset_name: DatasetName { pool_name: ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
8452024-09-23T18:20:43.043ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 2818b80d-465e-48ce-92dc-2436b4b704cd, dataset_name: DatasetName { pool_name: ZpoolName { id: 73f13013-70b2-4020-853f-5a70920c296f (zpool), kind: External }, kind: Crucible }, responder: ... }
8462024-09-23T18:20:43.112ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8472024-09-23T18:20:43.112ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-zones.json file = common/src/ledger.rs:177
8482024-09-23T18:20:43.113ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-zones.json file = common/src/ledger.rs:177
8492024-09-23T18:20:43.113ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8502024-09-23T18:20:43.214ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8512024-09-23T18:20:43.243ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8522024-09-23T18:20:43.244ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8532024-09-23T18:20:43.275ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8542024-09-23T18:20:43.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8552024-09-23T18:20:43.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8562024-09-23T18:20:43.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8572024-09-23T18:20:43.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8582024-09-23T18:20:43.275ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8592024-09-23T18:20:43.277ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8602024-09-23T18:20:43.524ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
8612024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8622024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8632024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8642024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8652024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8662024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8672024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8682024-09-23T18:20:43.524ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8692024-09-23T18:20:43.524ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712 file = illumos-utils/src/zone.rs:316
8702024-09-23T18:20:43.530ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905 file = illumos-utils/src/zone.rs:316
8712024-09-23T18:20:43.536ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349 file = illumos-utils/src/zone.rs:316
8722024-09-23T18:20:43.542ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c file = illumos-utils/src/zone.rs:316
8732024-09-23T18:20:43.547ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53 file = illumos-utils/src/zone.rs:316
8742024-09-23T18:20:43.552ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55 file = illumos-utils/src/zone.rs:316
8752024-09-23T18:20:43.741ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712 file = illumos-utils/src/zone.rs:354
8762024-09-23T18:20:43.747ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905 file = illumos-utils/src/zone.rs:354
8772024-09-23T18:20:43.753ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349 file = illumos-utils/src/zone.rs:354
8782024-09-23T18:20:43.758ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c file = illumos-utils/src/zone.rs:354
8792024-09-23T18:20:43.763ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53 file = illumos-utils/src/zone.rs:354
8802024-09-23T18:20:43.770ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55 file = illumos-utils/src/zone.rs:354
8812024-09-23T18:20:43.776ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee file = illumos-utils/src/zone.rs:316
8822024-09-23T18:20:43.781ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157 file = illumos-utils/src/zone.rs:316
8832024-09-23T18:20:43.786ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512 file = illumos-utils/src/zone.rs:316
8842024-09-23T18:20:43.792ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8 file = illumos-utils/src/zone.rs:316
8852024-09-23T18:20:43.798ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46 file = illumos-utils/src/zone.rs:316
8862024-09-23T18:20:43.804ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49 file = illumos-utils/src/zone.rs:316
8872024-09-23T18:20:43.811ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409 file = illumos-utils/src/zone.rs:316
8882024-09-23T18:20:43.817ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd file = illumos-utils/src/zone.rs:316
8892024-09-23T18:20:43.862ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
8902024-09-23T18:20:43.862ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee file = illumos-utils/src/zone.rs:354
8912024-09-23T18:20:43.869ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157 file = illumos-utils/src/zone.rs:354
8922024-09-23T18:20:43.874ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512 file = illumos-utils/src/zone.rs:354
8932024-09-23T18:20:43.880ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8 file = illumos-utils/src/zone.rs:354
8942024-09-23T18:20:43.886ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46 file = illumos-utils/src/zone.rs:354
8952024-09-23T18:20:43.895ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49 file = illumos-utils/src/zone.rs:354
8962024-09-23T18:20:43.901ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409 file = illumos-utils/src/zone.rs:354
8972024-09-23T18:20:43.908ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd file = illumos-utils/src/zone.rs:354
8982024-09-23T18:20:43.973ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
8992024-09-23T18:20:44.047ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9002024-09-23T18:20:44.119ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9012024-09-23T18:20:44.177ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
9022024-09-23T18:20:44.180ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9032024-09-23T18:20:44.180ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9042024-09-23T18:20:44.246ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 138, 147]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
9052024-09-23T18:20:44.246ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = external_dns
9062024-09-23T18:20:44.252ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9072024-09-23T18:20:44.252ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9082024-09-23T18:20:44.308ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 246, 214]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
9092024-09-23T18:20:44.308ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = external_dns
9102024-09-23T18:20:44.308ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
9112024-09-23T18:20:44.308ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
9122024-09-23T18:20:44.310ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9132024-09-23T18:20:44.310ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9142024-09-23T18:20:44.366ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 169, 46]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9152024-09-23T18:20:44.366ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
9162024-09-23T18:20:44.368ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9172024-09-23T18:20:44.368ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9182024-09-23T18:20:44.422ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 181, 224]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9192024-09-23T18:20:44.422ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
9202024-09-23T18:20:44.424ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
9212024-09-23T18:20:44.424ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
9222024-09-23T18:20:44.479ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 163, 135]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
9232024-09-23T18:20:44.480ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1261 zone_type = nexus
9242024-09-23T18:20:44.480ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
9252024-09-23T18:20:44.482ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9262024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
9272024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
9282024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
9292024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
9302024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
9312024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
9322024-09-23T18:20:44.548ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
9332024-09-23T18:20:44.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9342024-09-23T18:20:44.553ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
9352024-09-23T18:20:44.553ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
9362024-09-23T18:20:44.553ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
9372024-09-23T18:20:44.557ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
9382024-09-23T18:20:44.561ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9392024-09-23T18:20:44.561ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9402024-09-23T18:20:44.609ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
9412024-09-23T18:20:44.662ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e file = illumos-utils/src/zone.rs:316
9422024-09-23T18:20:44.804ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275 file = illumos-utils/src/zone.rs:316
9432024-09-23T18:20:44.823ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9442024-09-23T18:20:44.824ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e file = illumos-utils/src/zone.rs:316
9452024-09-23T18:20:44.836ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e file = illumos-utils/src/zone.rs:354
9462024-09-23T18:20:44.846ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7 file = illumos-utils/src/zone.rs:316
9472024-09-23T18:20:44.856ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275 file = illumos-utils/src/zone.rs:354
9482024-09-23T18:20:44.947ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e file = illumos-utils/src/zone.rs:354
9492024-09-23T18:20:44.961ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7 file = illumos-utils/src/zone.rs:354
9502024-09-23T18:20:45.052ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3 file = illumos-utils/src/zone.rs:316
9512024-09-23T18:20:45.104ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3 file = illumos-utils/src/zone.rs:354
9522024-09-23T18:20:45.876ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9532024-09-23T18:20:45.876ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
9542024-09-23T18:20:45.884ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9552024-09-23T18:20:45.884ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
9562024-09-23T18:20:45.931ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_67a20504-7be3-48ff-bb32-7f9411b1342b/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='bcb5244f-1914-4196-81eb-62059174a92d'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9572024-09-23T18:20:45.931ZINFOSledAgent (ServiceManager): Booting oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
9582024-09-23T18:20:45.979ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9592024-09-23T18:20:45.979ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
9602024-09-23T18:20:46.009ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e381d5a5-91f1-4b1c-b0ad-128e813a0a94/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='b3770a92-4277-4e99-84dd-7dd211670a9b'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9612024-09-23T18:20:46.010ZINFOSledAgent (ServiceManager): Booting oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
9622024-09-23T18:20:46.146ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_016c0356-f4b4-4d6b-8c24-edfd2951cfe3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='75ef87e2-fe1f-4890-91ec-4dbce800b633'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9632024-09-23T18:20:46.146ZINFOSledAgent (ServiceManager): Booting oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
9642024-09-23T18:20:46.154ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='43821df6-7a7b-49df-9a53-ebf281e00a55'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9652024-09-23T18:20:46.154ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
9662024-09-23T18:20:46.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 64.731092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
9672024-09-23T18:20:46.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 65.170761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
9682024-09-23T18:20:46.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 55.416974ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
9692024-09-23T18:20:46.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 55.204338ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
9702024-09-23T18:20:46.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 30.258956ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
9712024-09-23T18:20:46.266ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_17d503a6-b44e-42a3-ae74-5b19be755ad5/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='f4c3422d-e177-40e6-819c-ecd6d1a1df4e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9722024-09-23T18:20:46.266ZINFOSledAgent (ServiceManager): Booting oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
9732024-09-23T18:20:46.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 96.196047ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
9742024-09-23T18:20:46.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 147.932561ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
9752024-09-23T18:20:46.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 128.655621ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
9762024-09-23T18:20:46.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 55.595286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
9772024-09-23T18:20:46.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 58.21227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
9782024-09-23T18:20:46.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 175.565044ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
9792024-09-23T18:20:47.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 286.553804ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
9802024-09-23T18:20:47.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 234.068741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
9812024-09-23T18:20:47.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 125.13868ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
9822024-09-23T18:20:47.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 195.192351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
9832024-09-23T18:20:47.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 42.498085ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
9842024-09-23T18:20:47.493ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4edf146e-3ed8-4c23-b131-d4ed33864007/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='6b9d8a41-1986-4749-b075-cbd1c17a3ce9'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9852024-09-23T18:20:47.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 38.131429ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
9862024-09-23T18:20:47.619ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_b5c5f1b9-c752-4261-a49d-06cee1ff1f1a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='d15823b1-5b7a-4caf-a3e2-24f8807dc336'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9872024-09-23T18:20:47.620ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_1f7ede35-40db-4a1a-8087-fc87b6705700/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='b15914a0-2722-4a59-bb9b-4e74936ce4ec'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9882024-09-23T18:20:47.661ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c6321c72-f89d-471a-8986-1f1bf65a3de2/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='640d8cdd-8978-423c-9d12-bd064e591abe'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9892024-09-23T18:20:47.661ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_73f13013-70b2-4020-853f-5a70920c296f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='b335dd97-fdd9-473a-9fd7-00b78e6fae14'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9902024-09-23T18:20:47.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 332.910206ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
9912024-09-23T18:20:47.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 447.815732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
9922024-09-23T18:20:47.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 316.030648ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
9932024-09-23T18:20:47.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 289.90625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
9942024-09-23T18:20:47.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 511.027856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
9952024-09-23T18:20:47.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 111.319277ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
9962024-09-23T18:20:47.754ZINFOSledAgent (ServiceManager): Booting oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
9972024-09-23T18:20:47.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 35.147344ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
9982024-09-23T18:20:47.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 73.35971ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
9992024-09-23T18:20:47.785ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10002024-09-23T18:20:47.793ZINFOSledAgent (ServiceManager): Booting oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10012024-09-23T18:20:47.800ZINFOSledAgent (ServiceManager): Booting oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10022024-09-23T18:20:47.808ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10032024-09-23T18:20:47.828ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 98.924907ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10042024-09-23T18:20:47.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 187.169904ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
10052024-09-23T18:20:47.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 158.296941ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10062024-09-23T18:20:47.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 158.208986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10072024-09-23T18:20:48.026ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.010841304s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
10082024-09-23T18:20:48.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 972.853077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
10092024-09-23T18:20:48.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 958.241428ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
10102024-09-23T18:20:48.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 36.491635ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10112024-09-23T18:20:48.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 498.322909ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10122024-09-23T18:20:48.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 43.168621ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10132024-09-23T18:20:48.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 27.315079ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10142024-09-23T18:20:48.130ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 29.100769ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10152024-09-23T18:20:48.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 65.990366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10162024-09-23T18:20:48.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 287.804048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10172024-09-23T18:20:48.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 411.921512ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
10182024-09-23T18:20:48.186ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 136.614719ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10192024-09-23T18:20:48.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 85.891858ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10202024-09-23T18:20:48.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 131.605698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10212024-09-23T18:20:48.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 60.428232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10222024-09-23T18:20:48.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 770.870715ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
10232024-09-23T18:20:48.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 165.45222ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10242024-09-23T18:20:48.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 796.641593ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10252024-09-23T18:20:48.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 165.52104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10262024-09-23T18:20:48.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 215.708883ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10272024-09-23T18:20:48.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 290.072003ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10282024-09-23T18:20:48.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 118.837548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10292024-09-23T18:20:48.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 725.918614ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
10302024-09-23T18:20:48.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 792.682555ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10312024-09-23T18:20:48.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 464.850614ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10322024-09-23T18:20:48.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 737.198753ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
10332024-09-23T18:20:48.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 547.238371ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10342024-09-23T18:20:48.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 284.935662ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10352024-09-23T18:20:48.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 526.133826ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10362024-09-23T18:20:48.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 330.250369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10372024-09-23T18:20:49.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 1.464521479s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
10382024-09-23T18:20:49.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.127510111s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
10392024-09-23T18:20:49.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 1.267846409s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
10402024-09-23T18:20:49.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 837.459906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
10412024-09-23T18:20:49.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 265.425246ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10422024-09-23T18:20:49.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 828.414036ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10432024-09-23T18:20:49.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 995.263115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10442024-09-23T18:20:49.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 511.56603ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10452024-09-23T18:20:49.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 570.908835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10462024-09-23T18:20:49.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 691.192405ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10472024-09-23T18:20:49.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 857.924798ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
10482024-09-23T18:20:49.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 856.304759ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10492024-09-23T18:20:49.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 687.472662ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
10502024-09-23T18:20:49.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 793.648977ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10512024-09-23T18:20:49.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 846.754489ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10522024-09-23T18:20:49.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.247826535s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
10532024-09-23T18:20:49.932ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10542024-09-23T18:20:49.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 1.047396889s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10552024-09-23T18:20:49.951ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
10562024-09-23T18:20:49.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 777.698113ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10572024-09-23T18:20:50.077ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 529.364232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10582024-09-23T18:20:50.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 917.659569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
10592024-09-23T18:20:50.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.101164003s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
10602024-09-23T18:20:50.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 632.37611ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
10612024-09-23T18:20:50.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 584.142849ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10622024-09-23T18:20:50.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 1.409416514s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
10632024-09-23T18:20:50.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 541.693975ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10642024-09-23T18:20:50.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 32.557049ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
10652024-09-23T18:20:50.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 1.182185187s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
10662024-09-23T18:20:50.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 147.778391ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
10672024-09-23T18:20:50.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 814.908485ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
10682024-09-23T18:20:50.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 138.883205ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
10692024-09-23T18:20:50.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 600.367961ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10702024-09-23T18:20:50.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 911.965751ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
10712024-09-23T18:20:50.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 393.989215ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
10722024-09-23T18:20:50.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 804.449418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
10732024-09-23T18:20:50.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 675.136085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10742024-09-23T18:20:50.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 546.351763ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10752024-09-23T18:20:50.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 1.379451939s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
10762024-09-23T18:20:51.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 639.60639ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
10772024-09-23T18:20:51.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 604.813325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
10782024-09-23T18:20:51.146ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10792024-09-23T18:20:51.146ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
10802024-09-23T18:20:51.191ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.201035968s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
10812024-09-23T18:20:51.201ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10822024-09-23T18:20:51.201ZINFOSledAgent (ServiceManager): Booting oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
10832024-09-23T18:20:51.210ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
10842024-09-23T18:20:51.210ZINFOSledAgent (ServiceManager): Booting oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
10852024-09-23T18:20:51.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 1.060044282s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
10862024-09-23T18:20:51.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.313063858s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
10872024-09-23T18:20:51.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.481091211s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
10882024-09-23T18:20:51.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 777.739568ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
10892024-09-23T18:20:51.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 49.788919ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
10902024-09-23T18:20:51.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 66.150699ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
10912024-09-23T18:20:51.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 34.612875ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
10922024-09-23T18:20:51.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 108.757554ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
10932024-09-23T18:20:51.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 608.481719ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
10942024-09-23T18:20:51.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 119.779579ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
10952024-09-23T18:20:51.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 96.287923ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
10962024-09-23T18:20:51.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 1.116254715s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
10972024-09-23T18:20:51.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 241.606162ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
10982024-09-23T18:20:51.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.048440021s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
10992024-09-23T18:20:51.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 126.765546ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
11002024-09-23T18:20:51.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 235.032622ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11012024-09-23T18:20:51.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 1.082574462s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
11022024-09-23T18:20:51.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 797.375775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
11032024-09-23T18:20:51.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 1.427120794s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
11042024-09-23T18:20:51.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 1.200722752s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
11052024-09-23T18:20:51.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 1.473035498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
11062024-09-23T18:20:51.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 427.845543ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
11072024-09-23T18:20:51.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 424.912253ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
11082024-09-23T18:20:51.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 466.437764ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11092024-09-23T18:20:52.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 1.106564981s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
11102024-09-23T18:20:52.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 1.066049859s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
11112024-09-23T18:20:52.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 1.193534421s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
11122024-09-23T18:20:52.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 981.478304ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
11132024-09-23T18:20:52.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 573.144418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
11142024-09-23T18:20:52.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 1.027967681s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
11152024-09-23T18:20:52.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 426.080385ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11162024-09-23T18:20:52.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.243816477s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
11172024-09-23T18:20:52.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 1.364368314s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
11182024-09-23T18:20:52.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.137445184s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
11192024-09-23T18:20:52.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.464926647s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
11202024-09-23T18:20:52.702ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11212024-09-23T18:20:52.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 782.238494ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
11222024-09-23T18:20:52.708ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11232024-09-23T18:20:52.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 1.400039943s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
11242024-09-23T18:20:52.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 1.11080703s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11252024-09-23T18:20:52.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.34626447s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
11262024-09-23T18:20:52.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 959.210411ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
11272024-09-23T18:20:52.950ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 695.495338ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
11282024-09-23T18:20:53.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 44.0852ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11292024-09-23T18:20:53.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 110.054715ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11302024-09-23T18:20:53.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 915.434988ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
11312024-09-23T18:20:53.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 276.137553ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11322024-09-23T18:20:53.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 955.501872ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
11332024-09-23T18:20:53.238ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 1.046281885s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
11342024-09-23T18:20:53.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 1.025836515s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
11352024-09-23T18:20:53.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.340234747s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
11362024-09-23T18:20:53.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 907.419423ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
11372024-09-23T18:20:53.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 720.16195ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
11382024-09-23T18:20:53.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 373.186839ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11392024-09-23T18:20:53.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 1.336727727s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
11402024-09-23T18:20:53.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 631.953393ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
11412024-09-23T18:20:53.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.356629075s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
11422024-09-23T18:20:53.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.414896967s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
11432024-09-23T18:20:53.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 1.294032297s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
11442024-09-23T18:20:53.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 541.044097ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11452024-09-23T18:20:53.901ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 1.398182209s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
11462024-09-23T18:20:53.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 1.023590318s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11472024-09-23T18:20:54.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 1.417424405s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
11482024-09-23T18:20:54.133ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.15172538s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
11492024-09-23T18:20:54.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 1.42846135s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
11502024-09-23T18:20:54.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 1.402116701s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
11512024-09-23T18:20:54.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 678.578571ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
11522024-09-23T18:20:54.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.125512826s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
11532024-09-23T18:20:54.231ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11542024-09-23T18:20:54.231ZINFOSledAgent (ServiceManager): Booting oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11552024-09-23T18:20:54.294ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 598.182636ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
11562024-09-23T18:20:54.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 1.189028806s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
11572024-09-23T18:20:54.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 873.465197ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
11582024-09-23T18:20:54.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 776.41728ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
11592024-09-23T18:20:54.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 677.233067ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11602024-09-23T18:20:54.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 39.553787ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11612024-09-23T18:20:54.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 90.069156ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11622024-09-23T18:20:54.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.414073185s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
11632024-09-23T18:20:54.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 133.446675ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11642024-09-23T18:20:54.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 767.073212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
11652024-09-23T18:20:54.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 453.40195ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11662024-09-23T18:20:54.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 1.480365802s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
11672024-09-23T18:20:54.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 732.231856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
11682024-09-23T18:20:54.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 810.019748ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11692024-09-23T18:20:55.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.064120966s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
11702024-09-23T18:20:55.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 531.766414ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11712024-09-23T18:20:55.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 789.959674ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
11722024-09-23T18:20:55.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 812.499105ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
11732024-09-23T18:20:55.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.013621858s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
11742024-09-23T18:20:55.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 752.89114ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
11752024-09-23T18:20:55.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 529.008197ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
11762024-09-23T18:20:55.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 1.418236158s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
11772024-09-23T18:20:55.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.046113751s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11782024-09-23T18:20:55.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 686.815249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
11792024-09-23T18:20:55.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 897.545806ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
11802024-09-23T18:20:55.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 778.789504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
11812024-09-23T18:20:55.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 884.848883ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
11822024-09-23T18:20:55.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 1.371999252s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
11832024-09-23T18:20:55.636ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c") failed: Property not found. retry in 1.102078816s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
11842024-09-23T18:20:55.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 1.149083837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
11852024-09-23T18:20:55.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 1.392704671s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
11862024-09-23T18:20:55.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 1.310880545s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
11872024-09-23T18:20:55.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.448636179s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
11882024-09-23T18:20:55.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 1.096788633s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
11892024-09-23T18:20:55.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 800.957ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
11902024-09-23T18:20:56.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 1.376459864s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
11912024-09-23T18:20:56.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.381907731s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
11922024-09-23T18:20:56.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.017728161s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
11932024-09-23T18:20:56.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712") failed: Property not found. retry in 1.483862975s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
11942024-09-23T18:20:56.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 856.709633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
11952024-09-23T18:20:56.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 602.52767ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
11962024-09-23T18:20:56.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 596.143804ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
11972024-09-23T18:20:56.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55") failed: Property not found. retry in 884.878196ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
11982024-09-23T18:20:56.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.261553184s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
11992024-09-23T18:20:56.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 915.012818ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12002024-09-23T18:20:56.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 1.481155036s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
12012024-09-23T18:20:56.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 861.782466ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12022024-09-23T18:20:56.838ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 1.415578454s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
12032024-09-23T18:20:56.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 947.445527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
12042024-09-23T18:20:57.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 617.686129ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
12052024-09-23T18:20:57.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 760.874559ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
12062024-09-23T18:20:57.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 764.93606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
12072024-09-23T18:20:57.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 977.936093ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
12082024-09-23T18:20:57.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 1.379575816s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12092024-09-23T18:20:57.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.011609562s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
12102024-09-23T18:20:57.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 958.969865ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
12112024-09-23T18:20:57.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 885.246735ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
12122024-09-23T18:20:57.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 1.32631287s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
12132024-09-23T18:20:57.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 892.499704ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12142024-09-23T18:20:57.488ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 813.170682ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12152024-09-23T18:20:57.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 838.888901ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
12162024-09-23T18:20:57.679ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.939428625s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
12172024-09-23T18:20:57.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 528.600299ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12182024-09-23T18:20:57.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.040993153s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12192024-09-23T18:20:57.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 1.41787615s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
12202024-09-23T18:20:57.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 1.370873411s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
12212024-09-23T18:20:57.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 1.319070216s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
12222024-09-23T18:20:58.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 612.014825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
12232024-09-23T18:20:58.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 1.110781067s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
12242024-09-23T18:20:58.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.263768229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
12252024-09-23T18:20:58.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 840.756921ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
12262024-09-23T18:20:58.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 1.19115036s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12272024-09-23T18:20:58.271ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 1.176342354s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
12282024-09-23T18:20:58.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 969.173176ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
12292024-09-23T18:20:58.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 716.929553ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12302024-09-23T18:20:58.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 1.050050442s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12312024-09-23T18:20:58.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 506.91617ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12322024-09-23T18:20:58.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 913.008332ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
12332024-09-23T18:20:58.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.355077844s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
12342024-09-23T18:20:58.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 517.629669ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
12352024-09-23T18:20:58.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 586.944915ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12362024-09-23T18:20:59.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 707.163437ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12372024-09-23T18:20:59.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.17973099s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12382024-09-23T18:20:59.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.464608897s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
12392024-09-23T18:20:59.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349") failed: Property not found. retry in 863.159857ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
12402024-09-23T18:20:59.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 809.560673ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
12412024-09-23T18:20:59.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 604.483666ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
12422024-09-23T18:20:59.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 649.353874ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
12432024-09-23T18:20:59.289ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 821.452502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
12442024-09-23T18:20:59.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 904.589555ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
12452024-09-23T18:20:59.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 718.567034ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12462024-09-23T18:20:59.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 1.241529641s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12472024-09-23T18:20:59.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 997.906361ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12482024-09-23T18:20:59.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905") failed: Property not found. retry in 1.138304382s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
12492024-09-23T18:20:59.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53") failed: Property not found. retry in 1.056859268s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
12502024-09-23T18:20:59.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 758.034392ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
12512024-09-23T18:20:59.760ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 533.882097ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12522024-09-23T18:20:59.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8") failed: Property not found. retry in 874.871286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
12532024-09-23T18:20:59.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 1.082930817s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
12542024-09-23T18:21:00.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 692.573399ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
12552024-09-23T18:21:00.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 779.470521ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
12562024-09-23T18:21:00.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.269887923s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12572024-09-23T18:21:00.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409") failed: Property not found. retry in 1.289498006s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
12582024-09-23T18:21:00.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512") failed: Property not found. retry in 1.232745643s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
12592024-09-23T18:21:00.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 647.642606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
12602024-09-23T18:21:00.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.192862458s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12612024-09-23T18:21:00.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 1.007639718s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12622024-09-23T18:21:00.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 557.399852ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12632024-09-23T18:21:00.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.026420178s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
12642024-09-23T18:21:00.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 612.434555ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12652024-09-23T18:21:00.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 1.167396972s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
12662024-09-23T18:21:00.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.385250457s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
12672024-09-23T18:21:00.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157") failed: Property not found. retry in 1.069167749s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
12682024-09-23T18:21:01.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 530.347676ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12692024-09-23T18:21:01.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7") failed: Property not found. retry in 1.294424392s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
12702024-09-23T18:21:01.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 778.485052ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12712024-09-23T18:21:01.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 601.89458ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12722024-09-23T18:21:01.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.268227025s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12732024-09-23T18:21:01.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 932.304309ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12742024-09-23T18:21:01.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 640.912453ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12752024-09-23T18:21:01.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd") failed: Property not found. retry in 1.288792264s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
12762024-09-23T18:21:01.932ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 762.002961ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12772024-09-23T18:21:01.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46") failed: Property not found. retry in 639.407177ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
12782024-09-23T18:21:02.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275") failed: Property not found. retry in 1.347040582s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
12792024-09-23T18:21:02.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e") failed: Property not found. retry in 1.280799401s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12802024-09-23T18:21:02.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49") failed: Property not found. retry in 1.352260076s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
12812024-09-23T18:21:02.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.091210597s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12822024-09-23T18:21:02.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.439218726s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12832024-09-23T18:21:02.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 684.190138ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12842024-09-23T18:21:03.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 902.117857ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12852024-09-23T18:21:03.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 594.648745ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12862024-09-23T18:21:04.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e") failed: Property not found. retry in 1.380925818s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12872024-09-23T18:21:04.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 1.424845599s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12882024-09-23T18:21:04.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee") failed: Property not found. retry in 715.505811ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
12892024-09-23T18:21:05.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3") failed: Property not found. retry in 615.365742ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12902024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_501a8d46-a8ba-41dd-888b-350783c6dce3
12912024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_96a2bbcf-5e5b-4bf1-82f3-5b2051b02712
12922024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_external_dns_cd3b9879-6155-40d6-93a9-0c64a75a3e6e
12932024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_336574a9-87bb-450e-9ef7-3aa3e0e79905
12942024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_4c6fe4b9-aa70-4c11-a8cb-78727c3b2349
12952024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_pantry_6837578d-2f18-4d5d-82e1-b69d1cbf6f4c
12962024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_5d81eec0-e923-40e4-875c-abea1d4aed53
12972024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_oximeter_43821df6-7a7b-49df-9a53-ebf281e00a55
12982024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_external_dns_544f6393-b5b8-4964-b9b8-d8467761342e
12992024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_b9740b46-002a-4447-8a9d-58caea4925b7
13002024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_clickhouse_f141ec5c-de3f-4dea-aa70-9c7e0ad6beee
13012024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_d28c60ef-c4db-4c3e-8e22-baaa61f68157
13022024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_517b3055-a59f-450e-a349-fc1fc2615512
13032024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_46c88b0c-8be0-41ef-83a1-5caeaba47dc8
13042024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_b473542b-f56f-44f7-ac97-717a7c8f5b46
13052024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_44aeefad-3a65-438e-8790-1c7a579bbb49
13062024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_7c0b2a6b-4c9a-4820-a10b-e91676739409
13072024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_crucible_2818b80d-465e-48ce-92dc-2436b4b704cd
13082024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3279 zone = oxz_nexus_efc72cb3-831b-4b4a-acac-bb2cd9209275
13092024-09-23T18:21:06.221ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
13102024-09-23T18:21:06.222ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
13112024-09-23T18:21:06.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 24202760 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:51030 req_id = 6b5f87da-a8d5-40f8-8d18-9ea66f2b6f16 response_code = 204 uri = /omicron-zones
13122024-09-23T18:21:06.227ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
13132024-09-23T18:21:06.227ZINFOSledAgent (RSS): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
13142024-09-23T18:21:06.227ZINFOSledAgent (RSS): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
13152024-09-23T18:21:06.229ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
13162024-09-23T18:21:06.230ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:720
13172024-09-23T18:21:06.261ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
13182024-09-23T18:21:06.587ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13192024-09-23T18:21:06.717ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13202024-09-23T18:21:06.856ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13212024-09-23T18:21:07.070ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13222024-09-23T18:21:07.285ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13232024-09-23T18:21:07.433ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13242024-09-23T18:21:07.569ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.513286509s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
13252024-09-23T18:21:07.760ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13262024-09-23T18:21:08.203ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13272024-09-23T18:21:08.524ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13282024-09-23T18:21:08.785ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13292024-09-23T18:21:09.497ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13302024-09-23T18:21:10.175ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13312024-09-23T18:21:10.843ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13322024-09-23T18:21:11.907ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13332024-09-23T18:21:12.022ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64550
13342024-09-23T18:21:12.024ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1446 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64550 req_id = 2febf352-b18f-4800-a918-6bb5932a6d6b response_code = 200 uri = /baseboard
13352024-09-23T18:21:12.114ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 7.211034909s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
13362024-09-23T18:21:12.857ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13372024-09-23T18:21:13.748ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13382024-09-23T18:21:15.616ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13392024-09-23T18:21:16.612ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13402024-09-23T18:21:18.750ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13412024-09-23T18:21:19.357ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 17.825115503s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:424 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
13422024-09-23T18:21:22.021ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13432024-09-23T18:21:23.755ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13442024-09-23T18:21:26.916ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/84f7defa-11d2-40bb-a94d-3ddc88dcf69e/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:949
13452024-09-23T18:21:31.112ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "f15c68c3-4c5e-4ec3-9b9a-0f27229d156c", "content-length": "173", "date": "Mon, 23 Sep 2024 18:21:30 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"84f7defa-11d2-40bb-a94d-3ddc88dcf69e\"", request_id: "f15c68c3-4c5e-4ec3-9b9a-0f27229d156c" } file = sled-agent/src/rack_setup/service.rs:949
13462024-09-23T18:21:37.381ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "96876edc-9648-43e0-973b-73f7b8fbcf1b", "content-length": "173", "date": "Mon, 23 Sep 2024 18:21:36 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"6061f953-e31a-4a2a-80c5-755d0e0127f8\"", request_id: "96876edc-9648-43e0-973b-73f7b8fbcf1b" } file = sled-agent/src/rack_setup/service.rs:949
13472024-09-23T18:21:42.054ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51187
13482024-09-23T18:21:42.054ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 29 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51187 req_id = d2a1f806-e834-40c2-b1c2-8b12c157c463 response_code = 200 uri = /baseboard
13492024-09-23T18:21:45.963ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "7eb3c4b6-ff14-404c-aafb-3e97c7442ab2", "content-length": "173", "date": "Mon, 23 Sep 2024 18:21:45 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"6061f953-e31a-4a2a-80c5-755d0e0127f8\"", request_id: "7eb3c4b6-ff14-404c-aafb-3e97c7442ab2" } file = sled-agent/src/rack_setup/service.rs:949
13502024-09-23T18:21:57.603ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "98cb7ea7-dac3-4831-92bd-73ed9fb5e25c", "content-length": "173", "date": "Mon, 23 Sep 2024 18:21:57 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"6061f953-e31a-4a2a-80c5-755d0e0127f8\"", request_id: "98cb7ea7-dac3-4831-92bd-73ed9fb5e25c" } file = sled-agent/src/rack_setup/service.rs:949
13512024-09-23T18:21:58.008ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
13522024-09-23T18:21:58.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40340
13532024-09-23T18:21:58.102ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13542024-09-23T18:21:58.296ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 193777 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40340 req_id = 09befacb-5cf6-4243-8776-aaedf92fc901 response_code = 200 uri = /inventory
13552024-09-23T18:21:58.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13562024-09-23T18:21:58.297ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-zones.json file = common/src/ledger.rs:177
13572024-09-23T18:21:58.298ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-zones.json file = common/src/ledger.rs:177
13582024-09-23T18:21:58.298ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2465 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40340 req_id = 63936227-5492-40c8-9784-7be57b0d1c69 response_code = 200 uri = /omicron-zones
13592024-09-23T18:22:02.144ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58892
13602024-09-23T18:22:02.144ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58892 req_id = fb55f77f-20a9-46ce-a93c-d579d73acfe2 response_code = 200 uri = /v2p
13612024-09-23T18:22:02.151ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36570
13622024-09-23T18:22:02.151ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36570 req_id = d983e8bf-f41b-430e-a733-546733b1976a response_code = 200 uri = /vpc-routes
13632024-09-23T18:22:02.163ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45832
13642024-09-23T18:22:02.163ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45832 req_id = d62d3544-d5d5-40a1-9431-f0a8c5b68753 response_code = 200 uri = /vpc-routes
13652024-09-23T18:22:02.255ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2206 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36570 req_id = a2f52757-9ebf-42a0-9a5d-30af61e9b236 response_code = 204 uri = /vpc-routes
13662024-09-23T18:22:02.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45832 req_id = c2c84681-6530-4d4f-aa8f-96846586be7b response_code = 204 uri = /vpc-routes
13672024-09-23T18:22:02.307ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33038
13682024-09-23T18:22:02.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33038 req_id = 665f5516-4e5a-4c8e-b8b1-e0a5883493a0 response_code = 200 uri = /v2p
13692024-09-23T18:22:02.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42893
13702024-09-23T18:22:02.310ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42893 req_id = 0ed272bd-9d3c-4259-a5e5-60a564fe027d response_code = 200 uri = /vpc-routes
13712024-09-23T18:22:02.321ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 753 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:42893 req_id = 2f4d1cfa-4d65-457a-a09d-5298571add94 response_code = 204 uri = /vpc-routes
13722024-09-23T18:22:02.375ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48522
13732024-09-23T18:22:02.375ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 302 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48522 req_id = 8dbf2e05-b295-4fc0-b418-c46aaf23129e response_code = 200 uri = /v2p
13742024-09-23T18:22:07.328ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:53690 remote_addr = [fd00:1122:3344:101::d]:57162 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
13752024-09-23T18:22:07.350ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 21630 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = cb45a511-0092-4fea-ad12-3e8d2963df7e response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
13762024-09-23T18:22:08.365ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64275
13772024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
13782024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
13792024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
13802024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
13812024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
13822024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
13832024-09-23T18:22:08.366ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
13842024-09-23T18:22:08.366ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 917 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64275 req_id = c82d718c-cc10-436a-9a6e-3d99d709e27e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
13852024-09-23T18:22:08.368ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
13862024-09-23T18:22:08.516ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55711
13872024-09-23T18:22:08.517ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13882024-09-23T18:22:08.713ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 196462 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55711 req_id = eef1e0ff-65ad-4073-b784-113001ccd28b response_code = 200 uri = /inventory
13892024-09-23T18:22:08.713ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48189
13902024-09-23T18:22:08.717ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13912024-09-23T18:22:08.718ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/config/omicron-zones.json file = common/src/ledger.rs:177
13922024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
13932024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
13942024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
13952024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
13962024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
13972024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
13982024-09-23T18:22:08.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
13992024-09-23T18:22:08.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4650 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48189 req_id = ea8e0401-1794-48ef-a1dd-b03458d8635e response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14002024-09-23T18:22:08.718ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/config/omicron-zones.json file = common/src/ledger.rs:177
14012024-09-23T18:22:08.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4984 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55711 req_id = 8b6980fc-9959-4bff-abdc-f68938906ce6 response_code = 200 uri = /omicron-zones
14022024-09-23T18:22:10.023ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45745
14032024-09-23T18:22:10.023ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14042024-09-23T18:22:10.023ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14052024-09-23T18:22:10.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14062024-09-23T18:22:10.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14072024-09-23T18:22:10.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14082024-09-23T18:22:10.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14092024-09-23T18:22:10.024ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
14102024-09-23T18:22:10.024ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 351 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45745 req_id = 76275eff-edc8-4461-90f2-7533feb4ea0a response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14112024-09-23T18:22:10.057ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58996
14122024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
14132024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14142024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14152024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14162024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
14172024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
14182024-09-23T18:22:10.057ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
14192024-09-23T18:22:10.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 280 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58996 req_id = a6834904-72eb-408a-98c0-53a5a75ab6bf response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
14202024-09-23T18:22:12.086ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:38809
14212024-09-23T18:22:12.087ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1075 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:38809 req_id = 9758264a-f809-4e5f-827b-e8f9778c109a response_code = 200 uri = /baseboard
14222024-09-23T18:22:15.740ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57638
14232024-09-23T18:22:15.740ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57638 req_id = 0c8e196a-82aa-4dc2-98bf-d13e84a8f3d6 response_code = 200 uri = /vpc-routes
14242024-09-23T18:22:15.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57638 req_id = 7cd5ddea-87b8-40a2-9dcf-0abee5f51dec response_code = 204 uri = /vpc-routes
14252024-09-23T18:22:32.235ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43723
14262024-09-23T18:22:32.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43723 req_id = 9d796e20-6ec9-4ecd-ade1-a86c803672bf response_code = 200 uri = /v2p
14272024-09-23T18:22:32.298ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56040
14282024-09-23T18:22:32.298ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56040 req_id = 6f364223-a808-4e83-89a4-98151f7d52df response_code = 200 uri = /vpc-routes
14292024-09-23T18:22:32.301ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56366
14302024-09-23T18:22:32.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2266 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56366 req_id = 4b8790e1-26e3-4abf-94e6-62edd37358c9 response_code = 200 uri = /vpc-routes
14312024-09-23T18:22:32.316ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 227 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56040 req_id = a1af1b84-c15a-42ff-b377-68c67ea5ef65 response_code = 204 uri = /vpc-routes
14322024-09-23T18:22:32.317ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1285 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56366 req_id = bf81e521-e5ee-4869-919b-59d8ea679eab response_code = 204 uri = /vpc-routes
14332024-09-23T18:22:32.453ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38654
14342024-09-23T18:22:32.453ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38654 req_id = 65f8d047-f6d4-43ba-afa0-8bd5a7bab726 response_code = 200 uri = /vpc-routes
14352024-09-23T18:22:32.455ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52547
14362024-09-23T18:22:32.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1217 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52547 req_id = 287e9d95-14ac-44b0-98bf-c8d1e013680e response_code = 200 uri = /v2p
14372024-09-23T18:22:32.464ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38654 req_id = dcb719d8-61f7-413d-b9e7-371f87577be0 response_code = 204 uri = /vpc-routes
14382024-09-23T18:22:32.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41167
14392024-09-23T18:22:32.554ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
14402024-09-23T18:22:32.603ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
14412024-09-23T18:22:32.656ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102546 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41167 req_id = 87745afc-c788-43a2-8f14-00b9d642a6ac response_code = 204 uri = /switch-ports
14422024-09-23T18:22:32.656ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50178
14432024-09-23T18:22:32.657ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 224 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50178 req_id = 814e249d-0bfd-43f8-ac40-c6acf77a6686 response_code = 200 uri = /v2p
14442024-09-23T18:22:32.658ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1271 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41167 req_id = 388ac988-05c8-43c2-8e46-f8640ee27632 response_code = 200 uri = /network-bootstore-config
14452024-09-23T18:22:32.709ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61460
14462024-09-23T18:22:32.710ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
14472024-09-23T18:22:32.752ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
14482024-09-23T18:22:32.808ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98049 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61460 req_id = a82621d5-9deb-4373-ade5-1b7ce1f7064d response_code = 204 uri = /switch-ports
14492024-09-23T18:22:32.808ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42607
14502024-09-23T18:22:32.808ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
14512024-09-23T18:22:32.808ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
14522024-09-23T18:22:32.808ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f181c9e6-1009-477f-8ae5-05396e960261/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
14532024-09-23T18:22:32.814ZINFOSledAgent (bootstore): Writing ledger to /pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
14542024-09-23T18:22:32.815ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
14552024-09-23T18:22:32.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6971 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61460 req_id = a559c0ca-b743-4daa-bfe3-24203f14c5b3 response_code = 200 uri = /network-bootstore-config
14562024-09-23T18:22:32.815ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7275 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41167 req_id = ca7ea57c-a6f6-498d-94cd-49170ea62eea response_code = 204 uri = /network-bootstore-config
14572024-09-23T18:22:32.847ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
14582024-09-23T18:22:32.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92623 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42607 req_id = a71249a3-aeb6-48ec-8827-4bb33c5b6877 response_code = 204 uri = /switch-ports
14592024-09-23T18:22:32.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42607 req_id = 40378bd3-5cad-4ab6-b5a5-310374f05a2a response_code = 200 uri = /network-bootstore-config
14602024-09-23T18:22:37.335ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7938 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = 4c02245a-5ba7-4504-9417-cd854d4e3fee response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
14612024-09-23T18:22:42.119ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59451
14622024-09-23T18:22:42.119ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59451 req_id = dc4b2c6b-a6b4-4c82-9560-337892d3b3f4 response_code = 200 uri = /baseboard
14632024-09-23T18:23:02.204ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60531
14642024-09-23T18:23:02.205ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1192 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60531 req_id = 79f819a0-85d4-4d4b-94d2-2d90ac725441 response_code = 200 uri = /vpc-routes
14652024-09-23T18:23:02.222ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60531 req_id = 708a0af8-0ecb-489f-ae73-f8e5a5ec8eff response_code = 204 uri = /vpc-routes
14662024-09-23T18:23:02.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64289
14672024-09-23T18:23:02.257ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64289 req_id = f4f44722-6ea2-4810-9056-ccf118f2cbc7 response_code = 200 uri = /v2p
14682024-09-23T18:23:02.293ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63125
14692024-09-23T18:23:02.293ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63125 req_id = 3f847df2-0566-4ee3-bb33-408210c2796c response_code = 200 uri = /vpc-routes
14702024-09-23T18:23:02.305ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:63125 req_id = d71fafaa-4fea-45d9-bfec-7601d0e0b8f2 response_code = 204 uri = /vpc-routes
14712024-09-23T18:23:02.320ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61618
14722024-09-23T18:23:02.320ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
14732024-09-23T18:23:02.362ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
14742024-09-23T18:23:02.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 108550 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61618 req_id = fe045f12-7445-461d-8c22-f35850a1459c response_code = 204 uri = /switch-ports
14752024-09-23T18:23:02.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55094
14762024-09-23T18:23:02.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57978
14772024-09-23T18:23:02.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41413
14782024-09-23T18:23:02.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46716
14792024-09-23T18:23:02.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 395 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61618 req_id = e4991f43-4320-4b88-8ccf-63d971807ccb response_code = 200 uri = /network-bootstore-config
14802024-09-23T18:23:02.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 289 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57978 req_id = 38b6ab63-fe7d-47d1-9163-ff5838363dd5 response_code = 200 uri = /v2p
14812024-09-23T18:23:02.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 982 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55094 req_id = 0eac53ec-4d0b-495c-9c0e-03626232a5dc response_code = 200 uri = /vpc-routes
14822024-09-23T18:23:02.430ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
14832024-09-23T18:23:02.430ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1076 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46716 req_id = 48533f92-d1b8-48b8-8bd6-996b4fbecdd0 response_code = 200 uri = /v2p
14842024-09-23T18:23:02.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55094 req_id = 6ba974d1-50db-40cf-84fc-fb66a9ed5a71 response_code = 204 uri = /vpc-routes
14852024-09-23T18:23:02.463ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43037
14862024-09-23T18:23:02.470ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
14872024-09-23T18:23:02.528ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
14882024-09-23T18:23:02.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99444 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41413 req_id = e5f9afc7-6e0b-4105-b0bf-080145cc12bb response_code = 204 uri = /switch-ports
14892024-09-23T18:23:02.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 572 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41413 req_id = e03c866d-6332-4a10-b2be-afe5323eb8c9 response_code = 200 uri = /network-bootstore-config
14902024-09-23T18:23:02.562ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
14912024-09-23T18:23:02.616ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 153611 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43037 req_id = e3d2411c-ddc4-4e6a-b492-967468f55520 response_code = 204 uri = /switch-ports
14922024-09-23T18:23:02.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43037 req_id = 780630f0-6663-4b2b-9b0b-5dcf9ef0ac38 response_code = 200 uri = /network-bootstore-config
14932024-09-23T18:23:07.335ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7494 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = 6cab0579-8361-4349-8b31-4a877b33f2cf response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
14942024-09-23T18:23:12.154ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41832
14952024-09-23T18:23:12.154ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 37 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41832 req_id = 569c46bb-4a90-4bdf-a987-42fa6e335150 response_code = 200 uri = /baseboard
14962024-09-23T18:23:32.344ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35241
14972024-09-23T18:23:32.345ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35241 req_id = d4c2c4a9-001b-484d-b5b9-9574c883aff6 response_code = 200 uri = /vpc-routes
14982024-09-23T18:23:32.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35241 req_id = ea838930-df99-449a-8f6f-d8896ddfd4fb response_code = 204 uri = /vpc-routes
14992024-09-23T18:23:32.409ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52357
15002024-09-23T18:23:32.409ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 34 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52357 req_id = 43392bfb-1d45-47a6-8567-3e82f31cb404 response_code = 200 uri = /vpc-routes
15012024-09-23T18:23:32.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1058 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52357 req_id = 184164a2-02e2-43d9-b269-02582001b248 response_code = 204 uri = /vpc-routes
15022024-09-23T18:23:32.431ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64669
15032024-09-23T18:23:32.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64669 req_id = b381f907-9bfe-41dc-8eb8-013ef463c6f1 response_code = 200 uri = /vpc-routes
15042024-09-23T18:23:32.446ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1072 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64669 req_id = 919b4f03-31ea-4407-a436-91362b3e729c response_code = 204 uri = /vpc-routes
15052024-09-23T18:23:32.482ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61624
15062024-09-23T18:23:32.482ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61624 req_id = 8bd0d181-a108-420c-a7cc-e49467dc3273 response_code = 200 uri = /v2p
15072024-09-23T18:23:32.554ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54384
15082024-09-23T18:23:32.554ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 349 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54384 req_id = 6d971259-e04e-4b80-918a-77009551472d response_code = 200 uri = /v2p
15092024-09-23T18:23:32.567ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62135
15102024-09-23T18:23:32.567ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62135 req_id = e648397c-3753-4c32-88be-041f2db5bd0f response_code = 200 uri = /v2p
15112024-09-23T18:23:32.633ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51658
15122024-09-23T18:23:32.634ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15132024-09-23T18:23:32.672ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15142024-09-23T18:23:32.725ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91666 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51658 req_id = 45b9cc43-c5c8-4638-9718-c16d0d842c56 response_code = 204 uri = /switch-ports
15152024-09-23T18:23:32.725ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46897
15162024-09-23T18:23:32.725ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45689
15172024-09-23T18:23:32.726ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 301 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51658 req_id = 6054317f-2078-4455-acbb-6f9af2b1110b response_code = 200 uri = /network-bootstore-config
15182024-09-23T18:23:32.726ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15192024-09-23T18:23:32.764ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15202024-09-23T18:23:32.819ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15212024-09-23T18:23:32.819ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93024 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45689 req_id = 3e33e626-938b-40af-940a-8551e848ccc2 response_code = 204 uri = /switch-ports
15222024-09-23T18:23:32.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 569 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45689 req_id = b87fb222-88a6-4b71-af24-3f51ebde924d response_code = 200 uri = /network-bootstore-config
15232024-09-23T18:23:32.859ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15242024-09-23T18:23:32.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 184140 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46897 req_id = b7c644ff-a17e-4b03-a993-c6bbd7ecf9ac response_code = 204 uri = /switch-ports
15252024-09-23T18:23:32.910ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 173 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46897 req_id = ed0c20a1-65fe-408a-aa16-96bafa17e435 response_code = 200 uri = /network-bootstore-config
15262024-09-23T18:23:37.335ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8210 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = f95de2ad-6d6c-4ec3-9a0d-5c3f5ba5d682 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
15272024-09-23T18:23:42.187ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46202
15282024-09-23T18:23:42.187ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46202 req_id = c1ffbf22-bb14-455b-9484-f5e73777c7b8 response_code = 200 uri = /baseboard
15292024-09-23T18:24:02.149ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53696
15302024-09-23T18:24:02.149ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53696 req_id = cf3f7b8d-a0a4-438b-b566-1a0adb233465 response_code = 200 uri = /vpc-routes
15312024-09-23T18:24:02.161ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53696 req_id = 0dc82da0-4132-4d86-a41d-b93b04d860ec response_code = 204 uri = /vpc-routes
15322024-09-23T18:24:02.252ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50031
15332024-09-23T18:24:02.253ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15342024-09-23T18:24:02.293ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15352024-09-23T18:24:02.359ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106880 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50031 req_id = 05480fd2-54ff-4b7b-a496-096ae9b5617b response_code = 204 uri = /switch-ports
15362024-09-23T18:24:02.359ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39636
15372024-09-23T18:24:02.359ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55833
15382024-09-23T18:24:02.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 322 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50031 req_id = e641a72e-def3-41b8-b76e-03cd50f55ff0 response_code = 200 uri = /network-bootstore-config
15392024-09-23T18:24:02.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39636 req_id = 34c79140-0b81-4ed8-bf02-93c50f0d6814 response_code = 200 uri = /vpc-routes
15402024-09-23T18:24:02.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 326 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55833 req_id = d215b255-195e-4434-9867-9078dc2e6843 response_code = 200 uri = /v2p
15412024-09-23T18:24:02.371ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 987 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39636 req_id = 5f0500db-de6b-4874-80d8-7a7118c6c508 response_code = 204 uri = /vpc-routes
15422024-09-23T18:24:02.389ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62144
15432024-09-23T18:24:02.390ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62144 req_id = ca63dc51-9d28-4861-87e0-ad016d8f533a response_code = 200 uri = /v2p
15442024-09-23T18:24:02.393ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43545
15452024-09-23T18:24:02.393ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 20 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43545 req_id = 5fd33f6b-3b02-4b98-b83a-abc849c536b8 response_code = 200 uri = /vpc-routes
15462024-09-23T18:24:02.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43545 req_id = d28a9f6c-57b6-45d7-8369-a53b1d533a29 response_code = 204 uri = /vpc-routes
15472024-09-23T18:24:02.403ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35301
15482024-09-23T18:24:02.403ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15492024-09-23T18:24:02.442ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15502024-09-23T18:24:02.503ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100104 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35301 req_id = 5bb9e408-df92-423f-83e5-d19f5f476e3c response_code = 204 uri = /switch-ports
15512024-09-23T18:24:02.503ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62014
15522024-09-23T18:24:02.503ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64935
15532024-09-23T18:24:02.503ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 282 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35301 req_id = 5b0fedd1-8eff-45d5-875c-22c8a11959bb response_code = 200 uri = /network-bootstore-config
15542024-09-23T18:24:02.503ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64935 req_id = e20ac35b-8933-4a64-a161-7edb36718e61 response_code = 200 uri = /v2p
15552024-09-23T18:24:02.504ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15562024-09-23T18:24:02.540ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15572024-09-23T18:24:02.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91726 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62014 req_id = 6eefa663-3fa2-4585-ba45-6a092c1191d2 response_code = 204 uri = /switch-ports
15582024-09-23T18:24:02.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62014 req_id = 1dadf1b8-6338-406f-8952-6033bebc9af1 response_code = 200 uri = /network-bootstore-config
15592024-09-23T18:24:07.341ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5911 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = e551011e-062e-4de2-9d4c-2e659d1a1143 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
15602024-09-23T18:24:12.221ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35908
15612024-09-23T18:24:12.221ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35908 req_id = 0e492012-6427-4bfa-8fd1-05c85834d66e response_code = 200 uri = /baseboard
15622024-09-23T18:24:29.296ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
15632024-09-23T18:24:32.261ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41070
15642024-09-23T18:24:32.261ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41070 req_id = ae8cba53-2259-47ef-a1ca-6cb83e4c5fc5 response_code = 200 uri = /vpc-routes
15652024-09-23T18:24:32.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1488 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41070 req_id = 804343a6-c1b1-4f72-ad05-fc1947b2cd15 response_code = 204 uri = /vpc-routes
15662024-09-23T18:24:32.325ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59605
15672024-09-23T18:24:32.325ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59605 req_id = eb98611e-7c30-4885-885c-ba791b9cdaf7 response_code = 200 uri = /vpc-routes
15682024-09-23T18:24:32.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59605 req_id = bd3f5699-6b56-49d7-b134-3a15b7f2f564 response_code = 204 uri = /vpc-routes
15692024-09-23T18:24:32.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57863
15702024-09-23T18:24:32.431ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15712024-09-23T18:24:32.476ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15722024-09-23T18:24:32.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106681 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57863 req_id = 0e823a27-5a32-46db-bbea-a8b06499000e response_code = 204 uri = /switch-ports
15732024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44361
15742024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45737
15752024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35565
15762024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33921
15772024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40310
15782024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 357 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57863 req_id = 853cc6e8-31bb-4d9a-bde4-8fd65505e8cd response_code = 200 uri = /network-bootstore-config
15792024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33921 req_id = b883f1b2-b756-4345-b47f-dc7ddb14beba response_code = 200 uri = /vpc-routes
15802024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 305 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44361 req_id = 42fbcf3d-153d-4f0f-87fb-630a29bb8d65 response_code = 200 uri = /v2p
15812024-09-23T18:24:32.537ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15822024-09-23T18:24:32.537ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 337 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45737 req_id = b99ebf80-3a6c-45e7-80f4-2af4db83fd5e response_code = 200 uri = /v2p
15832024-09-23T18:24:32.544ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1549 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35565 req_id = 44734bec-1447-4b1a-bfd9-88a2b9ac9dc8 response_code = 200 uri = /v2p
15842024-09-23T18:24:32.547ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33921 req_id = 9432cdae-3950-44b9-9513-c9fcffd6c135 response_code = 204 uri = /vpc-routes
15852024-09-23T18:24:32.574ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15862024-09-23T18:24:32.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89120 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40310 req_id = 263b84e0-0d77-4609-94d3-eb492d928928 response_code = 204 uri = /switch-ports
15872024-09-23T18:24:32.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40310 req_id = 2094aa6f-bd8b-4f11-993f-3ff190d5c18f response_code = 200 uri = /network-bootstore-config
15882024-09-23T18:24:32.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43714
15892024-09-23T18:24:32.733ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
15902024-09-23T18:24:32.767ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
15912024-09-23T18:24:32.818ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 85853 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:43714 req_id = ca49da6e-16aa-403c-9bc2-755f91455f4c response_code = 204 uri = /switch-ports
15922024-09-23T18:24:32.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43714 req_id = 332aedd3-3e02-407f-a5b9-a58449d61a83 response_code = 200 uri = /network-bootstore-config
15932024-09-23T18:24:37.388ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7894 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = 731641cf-f3e5-40a2-9bbc-20aba4b7fbc7 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
15942024-09-23T18:24:42.253ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40606
15952024-09-23T18:24:42.254ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 25 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40606 req_id = 4fc7d557-a801-4f1a-870c-17940ad28333 response_code = 200 uri = /baseboard
15962024-09-23T18:25:02.124ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60913
15972024-09-23T18:25:02.125ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60913 req_id = d59446e0-baf1-4817-9f72-8ca72deae7c5 response_code = 200 uri = /vpc-routes
15982024-09-23T18:25:02.136ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60913 req_id = fa217343-8df1-4b90-856e-17af5e788681 response_code = 204 uri = /vpc-routes
15992024-09-23T18:25:02.214ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50935
16002024-09-23T18:25:02.215ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 974 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50935 req_id = 93613999-d589-4d4e-b269-b4d8b4113b53 response_code = 200 uri = /vpc-routes
16012024-09-23T18:25:02.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50935 req_id = 46781c01-9d9c-4006-aefc-cf2edad9b04e response_code = 204 uri = /vpc-routes
16022024-09-23T18:25:02.260ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53917
16032024-09-23T18:25:02.260ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16042024-09-23T18:25:02.300ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16052024-09-23T18:25:02.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101382 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:53917 req_id = c868f6b3-ed04-45b6-b72d-75fc8df1813d response_code = 204 uri = /switch-ports
16062024-09-23T18:25:02.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38957
16072024-09-23T18:25:02.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46035
16082024-09-23T18:25:02.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35821
16092024-09-23T18:25:02.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38957 req_id = 4d5804bd-3215-47d6-ae70-2eea72373323 response_code = 200 uri = /v2p
16102024-09-23T18:25:02.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46035 req_id = 12b606f9-1cc7-4481-b969-ca75f16de236 response_code = 200 uri = /v2p
16112024-09-23T18:25:02.363ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 400 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53917 req_id = f5cd7184-a207-4524-9073-91a213eee9b2 response_code = 200 uri = /network-bootstore-config
16122024-09-23T18:25:02.363ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35821 req_id = aba6a318-3466-4366-a6cf-1a371cdf7969 response_code = 200 uri = /vpc-routes
16132024-09-23T18:25:02.373ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35821 req_id = 6e5245a8-5c34-420d-b944-b18661b5fd04 response_code = 204 uri = /vpc-routes
16142024-09-23T18:25:02.379ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35018
16152024-09-23T18:25:02.379ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16162024-09-23T18:25:02.421ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16172024-09-23T18:25:02.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93806 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35018 req_id = dc9c8940-728f-414f-9a4e-83d4267e155f response_code = 204 uri = /switch-ports
16182024-09-23T18:25:02.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42137
16192024-09-23T18:25:02.473ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 210 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35018 req_id = 123ce9e2-4a95-4f1c-b2f8-4cf30c772f19 response_code = 200 uri = /network-bootstore-config
16202024-09-23T18:25:02.473ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16212024-09-23T18:25:02.480ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44867
16222024-09-23T18:25:02.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 341 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44867 req_id = 7842c11b-1a81-4ddb-8531-04e0b3bf7038 response_code = 200 uri = /v2p
16232024-09-23T18:25:02.512ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16242024-09-23T18:25:02.565ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92201 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42137 req_id = 0b6e5691-a278-4077-b79c-56e02876db4a response_code = 204 uri = /switch-ports
16252024-09-23T18:25:02.566ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 833 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42137 req_id = 9fe83eac-0f91-4127-8525-aeb3bdb04813 response_code = 200 uri = /network-bootstore-config
16262024-09-23T18:25:07.387ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8411 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = fa28a4f8-9a77-4974-9d37-74b234f3b7b1 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
16272024-09-23T18:25:12.287ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41731
16282024-09-23T18:25:12.287ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41731 req_id = d1aaa406-6cae-4967-81e6-2d8244f68abf response_code = 200 uri = /baseboard
16292024-09-23T18:25:32.326ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50028
16302024-09-23T18:25:32.326ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50028 req_id = 61531e03-c7b7-4742-9228-5f666f9eb9e7 response_code = 200 uri = /vpc-routes
16312024-09-23T18:25:32.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50162
16322024-09-23T18:25:32.335ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 276 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50162 req_id = eb9c06fc-b265-44bf-9caf-2b5ab482afa9 response_code = 200 uri = /vpc-routes
16332024-09-23T18:25:32.359ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 225 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50028 req_id = 5f072ff6-0cb3-4a6b-81fd-fc95c11a3aa0 response_code = 204 uri = /vpc-routes
16342024-09-23T18:25:32.361ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50162 req_id = 64b579f0-f619-4ef8-b854-d8dcefe3ee34 response_code = 204 uri = /vpc-routes
16352024-09-23T18:25:32.464ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33884
16362024-09-23T18:25:32.464ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 74 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33884 req_id = 66973df8-2afa-4138-b5bc-333ad8c6f8f4 response_code = 200 uri = /vpc-routes
16372024-09-23T18:25:32.479ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33884 req_id = ab854d8e-b3d9-4bf3-b6e6-6848efcd623f response_code = 204 uri = /vpc-routes
16382024-09-23T18:25:32.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49456
16392024-09-23T18:25:32.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49456 req_id = 3d929761-a0e0-480f-89d4-5a93ed1f252c response_code = 200 uri = /v2p
16402024-09-23T18:25:32.625ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62917
16412024-09-23T18:25:32.626ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 910 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62917 req_id = c19f342d-d04a-489f-a670-ef205281bfc6 response_code = 200 uri = /v2p
16422024-09-23T18:25:32.628ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34463
16432024-09-23T18:25:32.629ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16442024-09-23T18:25:32.673ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16452024-09-23T18:25:32.730ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101719 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34463 req_id = 7fcb9660-1887-46e7-810b-85221772c4da response_code = 204 uri = /switch-ports
16462024-09-23T18:25:32.730ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62565
16472024-09-23T18:25:32.731ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16482024-09-23T18:25:32.764ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16492024-09-23T18:25:32.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 90316 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:62565 req_id = 72a918e0-6d87-444d-9f2a-bead750726ef response_code = 204 uri = /switch-ports
16502024-09-23T18:25:32.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34463 req_id = 635b4660-4382-4fac-bd77-7c938e9d44b0 response_code = 200 uri = /network-bootstore-config
16512024-09-23T18:25:32.821ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61736
16522024-09-23T18:25:32.821ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53751
16532024-09-23T18:25:32.821ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 286 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62565 req_id = dbfbe112-7030-4be3-8c14-856c357d58c3 response_code = 200 uri = /network-bootstore-config
16542024-09-23T18:25:32.821ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16552024-09-23T18:25:32.822ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 225 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53751 req_id = a5059168-d2fc-4769-92cd-85c120642d02 response_code = 200 uri = /v2p
16562024-09-23T18:25:32.865ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16572024-09-23T18:25:32.925ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103919 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:61736 req_id = e4e57144-215a-460e-bcbe-3ecbbb02f799 response_code = 204 uri = /switch-ports
16582024-09-23T18:25:32.927ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 975 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61736 req_id = f9da2cf6-9931-4cf8-bb8f-ac4ca6d90ada response_code = 200 uri = /network-bootstore-config
16592024-09-23T18:25:37.388ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8570 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = cce58c8e-3e48-43e2-adbe-fdae766fd9b0 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
16602024-09-23T18:25:42.320ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37001
16612024-09-23T18:25:42.320ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37001 req_id = 83276335-78da-4bf2-81f7-3ee62a7d6b91 response_code = 200 uri = /baseboard
16622024-09-23T18:26:02.210ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43920
16632024-09-23T18:26:02.211ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 988 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43920 req_id = be47b530-3406-41aa-b0c3-200f22fe8e59 response_code = 200 uri = /vpc-routes
16642024-09-23T18:26:02.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43920 req_id = a2716a06-6259-4c46-94a8-bc5bd4e34c36 response_code = 204 uri = /vpc-routes
16652024-09-23T18:26:02.250ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64675
16662024-09-23T18:26:02.250ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64675 req_id = f1f4e5d5-2450-4d77-8b31-594f7dd13ba4 response_code = 200 uri = /vpc-routes
16672024-09-23T18:26:02.264ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64675 req_id = 1400d98a-dc4a-43dc-9085-cb163e6dedff response_code = 204 uri = /vpc-routes
16682024-09-23T18:26:02.265ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44655
16692024-09-23T18:26:02.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44655 req_id = c12a0b84-1c8d-4035-97ba-56894773b37e response_code = 200 uri = /v2p
16702024-09-23T18:26:02.313ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64864
16712024-09-23T18:26:02.313ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16722024-09-23T18:26:02.357ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16732024-09-23T18:26:02.424ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110452 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:64864 req_id = 275768b5-5812-478a-ae9c-902e62007f69 response_code = 204 uri = /switch-ports
16742024-09-23T18:26:02.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35300
16752024-09-23T18:26:02.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58055
16762024-09-23T18:26:02.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55884
16772024-09-23T18:26:02.424ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 286 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64864 req_id = e8256b87-0d91-4016-a562-2cf3a4b21cdd response_code = 200 uri = /network-bootstore-config
16782024-09-23T18:26:02.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 723 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55884 req_id = 388c414e-2172-42f1-a3c6-7a1f60f83a8e response_code = 200 uri = /vpc-routes
16792024-09-23T18:26:02.425ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16802024-09-23T18:26:02.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 943 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58055 req_id = 4d1f5929-9a9f-4159-b450-26f70bda1e9a response_code = 200 uri = /v2p
16812024-09-23T18:26:02.433ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 158 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55884 req_id = ea739067-4e4a-45ab-957f-f932a0d993eb response_code = 204 uri = /vpc-routes
16822024-09-23T18:26:02.462ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38351
16832024-09-23T18:26:02.462ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38351 req_id = 8cbf0229-bf6e-48e8-9075-71166d209f04 response_code = 200 uri = /v2p
16842024-09-23T18:26:02.463ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16852024-09-23T18:26:02.508ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63408
16862024-09-23T18:26:02.515ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
16872024-09-23T18:26:02.515ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 91094 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35300 req_id = b40c1f90-62db-4d82-981f-e387664af23b response_code = 204 uri = /switch-ports
16882024-09-23T18:26:02.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 655 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35300 req_id = 21724140-ce40-4732-82fd-ed7744a684a5 response_code = 200 uri = /network-bootstore-config
16892024-09-23T18:26:02.555ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
16902024-09-23T18:26:02.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101233 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63408 req_id = aaf61044-31d0-4c9f-a77e-b08c94196736 response_code = 204 uri = /switch-ports
16912024-09-23T18:26:02.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 671 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63408 req_id = 9a041514-8738-4125-b177-07d0d6896898 response_code = 200 uri = /network-bootstore-config
16922024-09-23T18:26:07.387ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7274 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = d7e38f22-eb84-45c7-af74-2c6e9cc51eba response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
16932024-09-23T18:26:12.354ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:62548
16942024-09-23T18:26:12.354ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:62548 req_id = aa887a5d-3765-4f46-9aaa-ed4ef386db74 response_code = 200 uri = /baseboard
16952024-09-23T18:26:32.354ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34144
16962024-09-23T18:26:32.354ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34144 req_id = aad936d8-698b-482b-a405-0d527f34ce46 response_code = 200 uri = /vpc-routes
16972024-09-23T18:26:32.359ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44559
16982024-09-23T18:26:32.360ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1296 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44559 req_id = c46063c1-ff7d-48b0-aeca-5480d6f0466b response_code = 200 uri = /v2p
16992024-09-23T18:26:32.368ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34144 req_id = ba7e08ff-840b-47e4-b105-555d64473137 response_code = 204 uri = /vpc-routes
17002024-09-23T18:26:32.375ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61331
17012024-09-23T18:26:32.375ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61331 req_id = 2649a88a-e3a2-4ae1-849e-c0bd2134af1e response_code = 200 uri = /vpc-routes
17022024-09-23T18:26:32.385ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64585
17032024-09-23T18:26:32.385ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64585 req_id = 833ae464-2f4e-4605-b9d0-a3280566d7db response_code = 200 uri = /vpc-routes
17042024-09-23T18:26:32.453ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48244
17052024-09-23T18:26:32.453ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17062024-09-23T18:26:32.493ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17072024-09-23T18:26:32.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 116462 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48244 req_id = 3e6089e7-013c-4fcf-89b1-271afc08ab8c response_code = 204 uri = /switch-ports
17082024-09-23T18:26:32.570ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55979
17092024-09-23T18:26:32.570ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61792
17102024-09-23T18:26:32.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 320 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64585 req_id = 9d94718e-1fbf-4414-8e88-c6e08b452187 response_code = 204 uri = /vpc-routes
17112024-09-23T18:26:32.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 401 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61331 req_id = 1dcc6fa6-4bbc-4c86-a817-8a62f2ffdd05 response_code = 204 uri = /vpc-routes
17122024-09-23T18:26:32.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 401 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48244 req_id = d85543c6-22d8-48a2-8866-e1cbdf12f048 response_code = 200 uri = /network-bootstore-config
17132024-09-23T18:26:32.571ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 355 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55979 req_id = 738d0c8d-12b9-41c4-bd0d-2703e3eddb84 response_code = 200 uri = /v2p
17142024-09-23T18:26:32.572ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1552 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61792 req_id = d65b3ec5-23cb-4f2f-9dff-f1e6be5695a7 response_code = 200 uri = /v2p
17152024-09-23T18:26:32.577ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64130
17162024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17172024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17182024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17192024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17202024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17212024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17222024-09-23T18:26:32.577ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17232024-09-23T18:26:32.577ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:64130 req_id = d8258383-93e2-421e-a99a-53fa46cebdcb response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17242024-09-23T18:26:32.717ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53791
17252024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17262024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17272024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17282024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17292024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17302024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17312024-09-23T18:26:32.718ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17322024-09-23T18:26:32.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 306 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53791 req_id = e20d0fe7-acaf-4181-b529-4122b31e00a0 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17332024-09-23T18:26:32.725ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65194
17342024-09-23T18:26:32.725ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17352024-09-23T18:26:32.760ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17362024-09-23T18:26:32.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88440 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:65194 req_id = 19006492-902f-4303-8d88-0bb4c591a189 response_code = 204 uri = /switch-ports
17372024-09-23T18:26:32.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65194 req_id = 9ad758e4-720c-49e0-be52-801e4ee00780 response_code = 200 uri = /network-bootstore-config
17382024-09-23T18:26:32.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52902
17392024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 2568eb85-be4f-4181-99fb-e5a6daca1d15, kind: Service { id: 544f6393-b5b8-4964-b9b8-d8467761342e }, name: Name("external-dns-544f6393-b5b8-4964-b9b8-d8467761342e"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 138, 147])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: c7b23545-cfea-4a44-9d64-7daf6dd5e237, kind: Service { id: cd3b9879-6155-40d6-93a9-0c64a75a3e6e }, name: Name("external-dns-cd3b9879-6155-40d6-93a9-0c64a75a3e6e"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 246, 214])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 1c060c1a-e485-446f-ad75-ea0a2c0954a2, kind: Service { id: b9740b46-002a-4447-8a9d-58caea4925b7 }, name: Name("nexus-b9740b46-002a-4447-8a9d-58caea4925b7"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 169, 46])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: b1ac0105-3717-4508-bfe7-7cd6e531b026, kind: Service { id: 501a8d46-a8ba-41dd-888b-350783c6dce3 }, name: Name("nexus-501a8d46-a8ba-41dd-888b-350783c6dce3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 163, 135])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: dca9677b-8460-49c9-b9b1-314fd3e786e5, kind: Service { id: efc72cb3-831b-4b4a-acac-bb2cd9209275 }, name: Name("nexus-efc72cb3-831b-4b4a-acac-bb2cd9209275"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 181, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17402024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17412024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17422024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17432024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17442024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17452024-09-23T18:26:32.875ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17462024-09-23T18:26:32.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 841 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52902 req_id = ecacd06f-b225-43b8-9614-1e8dcf73c92a response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17472024-09-23T18:26:32.910ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43731
17482024-09-23T18:26:32.910ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17492024-09-23T18:26:32.950ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17502024-09-23T18:26:33.008ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97943 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43731 req_id = 80a70ffe-3631-43a6-8d5c-2e04d8bf22b4 response_code = 204 uri = /switch-ports
17512024-09-23T18:26:33.009ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43731 req_id = 59640a90-4869-40d8-801a-c02712640b89 response_code = 200 uri = /network-bootstore-config
17522024-09-23T18:26:37.386ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7344 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = d9649418-eb54-4f27-9d88-9a7e565ac3e1 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
17532024-09-23T18:26:42.389ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58923
17542024-09-23T18:26:42.389ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58923 req_id = 758fb359-7eb9-427d-9b6e-7516a2274c5b response_code = 200 uri = /baseboard
17552024-09-23T18:27:02.322ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50143
17562024-09-23T18:27:02.322ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50143 req_id = efa31af2-b341-4b65-bc3d-ff71fe8a0eed response_code = 200 uri = /vpc-routes
17572024-09-23T18:27:02.333ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:50143 req_id = c65f8817-b4b2-48ea-95e4-b19c72536825 response_code = 204 uri = /vpc-routes
17582024-09-23T18:27:02.339ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34414
17592024-09-23T18:27:02.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 27 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34414 req_id = fe853028-dcdd-42cf-9248-1bbdf710988c response_code = 200 uri = /vpc-routes
17602024-09-23T18:27:02.341ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42490
17612024-09-23T18:27:02.341ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42490 req_id = 53f8f242-7fde-4a0a-ad03-f0aca78361c1 response_code = 200 uri = /v2p
17622024-09-23T18:27:02.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 957 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34414 req_id = f20c2e9b-b628-4ec3-971b-6a3cef6a30b3 response_code = 204 uri = /vpc-routes
17632024-09-23T18:27:02.366ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57646
17642024-09-23T18:27:02.366ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57646 req_id = aceeae8e-74c4-4bba-8474-8ed95cff007f response_code = 200 uri = /vpc-routes
17652024-09-23T18:27:02.393ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34452
17662024-09-23T18:27:02.393ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17672024-09-23T18:27:02.435ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17682024-09-23T18:27:02.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102797 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34452 req_id = e151ffa9-b8e0-41e2-b0d0-15cdad2b4af6 response_code = 204 uri = /switch-ports
17692024-09-23T18:27:02.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57646 req_id = dd1eb065-e154-4564-9b75-c35c05ff648d response_code = 204 uri = /vpc-routes
17702024-09-23T18:27:02.496ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37275
17712024-09-23T18:27:02.496ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36908
17722024-09-23T18:27:02.496ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56933
17732024-09-23T18:27:02.496ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60371
17742024-09-23T18:27:02.497ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 291 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34452 req_id = 80504528-aafd-46f5-96a3-a9c77952ac0e response_code = 200 uri = /network-bootstore-config
17752024-09-23T18:27:02.498ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17762024-09-23T18:27:02.498ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1136 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36908 req_id = e004edc6-f3d8-4f54-9759-d9f7e8756b44 response_code = 200 uri = /v2p
17772024-09-23T18:27:02.498ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37275 req_id = 6e36e0de-c43f-42d8-a893-8deb2267fa56 response_code = 200 uri = /v2p
17782024-09-23T18:27:02.535ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17792024-09-23T18:27:02.590ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
17802024-09-23T18:27:02.590ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93242 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60371 req_id = 97479192-ffc7-4c2a-87c9-6b6236f73b06 response_code = 204 uri = /switch-ports
17812024-09-23T18:27:02.595ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 559 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60371 req_id = de1b3811-8402-4ce6-9cce-34c91708e012 response_code = 200 uri = /network-bootstore-config
17822024-09-23T18:27:02.625ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
17832024-09-23T18:27:02.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 186971 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56933 req_id = 91463486-cb2f-4fcb-9206-febd5862a0c3 response_code = 204 uri = /switch-ports
17842024-09-23T18:27:02.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56933 req_id = 44a97b60-a731-4139-ae9e-4ccf7cdd1d3d response_code = 200 uri = /network-bootstore-config
17852024-09-23T18:27:07.502ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7107 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = 4f20cb2f-d5ab-4bc6-9f41-d4a8ef0b29d0 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
17862024-09-23T18:27:12.423ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45833
17872024-09-23T18:27:12.423ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45833 req_id = 5901f200-b09b-4db9-80d7-bb566d111912 response_code = 200 uri = /baseboard
17882024-09-23T18:27:32.320ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63072
17892024-09-23T18:27:32.320ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63072 req_id = 257bf0f0-c994-4499-9294-695d287e71d2 response_code = 200 uri = /vpc-routes
17902024-09-23T18:27:32.357ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52014
17912024-09-23T18:27:32.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52014 req_id = 145b8ac2-33c7-4613-960e-072e9a65e9d6 response_code = 200 uri = /vpc-routes
17922024-09-23T18:27:32.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52014 req_id = fec0acd3-c61b-49f5-8b53-7a1d9cbf4b28 response_code = 204 uri = /vpc-routes
17932024-09-23T18:27:32.395ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47234
17942024-09-23T18:27:32.395ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47234 req_id = d53c4999-7069-4768-b4fc-083d4f81bb2d response_code = 200 uri = /vpc-routes
17952024-09-23T18:27:32.401ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49463
17962024-09-23T18:27:32.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1062 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49463 req_id = 195c2433-6e50-4e94-9c7f-94984dbe25e3 response_code = 200 uri = /v2p
17972024-09-23T18:27:32.404ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63072 req_id = e8aceaf9-bfad-4a54-abb7-b74f3d6739d5 response_code = 204 uri = /vpc-routes
17982024-09-23T18:27:32.467ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59941
17992024-09-23T18:27:32.468ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59941 req_id = 7f6a913c-fcfa-42a7-8f84-b9a0cb573a0a response_code = 200 uri = /v2p
18002024-09-23T18:27:32.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47234 req_id = 86b87337-70d5-4171-8967-8b5fc18fb3e0 response_code = 204 uri = /vpc-routes
18012024-09-23T18:27:32.508ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45587
18022024-09-23T18:27:32.509ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18032024-09-23T18:27:32.547ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18042024-09-23T18:27:32.601ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93442 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:45587 req_id = 39742576-986c-49ca-bf63-8f8cb32a386e response_code = 204 uri = /switch-ports
18052024-09-23T18:27:32.602ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54535
18062024-09-23T18:27:32.602ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39467
18072024-09-23T18:27:32.602ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54647
18082024-09-23T18:27:32.602ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39467 req_id = c8353a5e-5cf9-4501-9ddd-620c0f1583c3 response_code = 200 uri = /v2p
18092024-09-23T18:27:32.603ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18102024-09-23T18:27:32.603ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1201 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45587 req_id = 38fe341e-29f2-4121-8bcd-7c20d9740f6d response_code = 200 uri = /network-bootstore-config
18112024-09-23T18:27:32.641ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18122024-09-23T18:27:32.696ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18132024-09-23T18:27:32.696ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94560 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54647 req_id = f0bbea10-9e95-4cf4-bed1-828aa490ebeb response_code = 204 uri = /switch-ports
18142024-09-23T18:27:32.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 572 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54647 req_id = 4279583a-22e4-4824-a1f9-eee61750abd1 response_code = 200 uri = /network-bootstore-config
18152024-09-23T18:27:32.730ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18162024-09-23T18:27:32.782ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 179826 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:54535 req_id = 8e381b5e-6f3b-48ad-bef8-537a9d25c031 response_code = 204 uri = /switch-ports
18172024-09-23T18:27:32.782ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54535 req_id = fed0debb-d4ad-4452-a5e9-d0f9e6428a65 response_code = 200 uri = /network-bootstore-config
18182024-09-23T18:27:37.503ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7632 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = d7d6b48f-bda2-49a3-8b8e-902a957be843 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
18192024-09-23T18:27:42.456ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:40764
18202024-09-23T18:27:42.456ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:40764 req_id = 80b2e080-864e-4c61-bf96-042bb17c8741 response_code = 200 uri = /baseboard
18212024-09-23T18:27:59.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47636
18222024-09-23T18:27:59.302ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47636 req_id = f7868c53-a32e-4abb-a1f4-976201f6296e response_code = 200 uri = /v2p
18232024-09-23T18:27:59.302ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(12082344) }
18242024-09-23T18:27:59.302ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47636 req_id = 6fd788f8-5001-4b46-8be2-33dbbcc888b6 response_code = 204 uri = /v2p
18252024-09-23T18:27:59.443ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62314
18262024-09-23T18:27:59.446ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:539 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(12082344))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-a9ed00178c9", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 43d3e8c1-1c9d-4f8d-859f-afdc043bed22, read_only_parent: Some(Volume { block_size: 512, id: 84a74b6e-69f6-442c-bbfb-9b80084c8e3d, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 1785d715-ae6b-4e8d-87eb-34c27f37d857, key: Some("wXAwZi9nltTjbd/msPIFqjBWLowdXPFrx1mZQRJyiA4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 43d3e8c1-1c9d-4f8d-859f-afdc043bed22, key: Some("M6UZwzKZo4nZdYMPs3lqJt+1Jk3PNYWcplgk4xlfk/Y="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::1a]:19000"] } }] } }], cloud_init_bytes: Some(..) } instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 metadata = InstanceMetadata { silo_id: 69506965-f491-446d-8149-f3af02e1bf91, project_id: 8eb631ba-e928-44bf-a2a0-24c9e291584b } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-23T18:27:59.443289850Z }
18272024-09-23T18:27:59.446ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:572 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 migration_id = None propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18282024-09-23T18:27:59.446ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1008 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 migration_id = None propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(12082344))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 147d34a2-4be8-4c4a-8765-e4bdba4c276a, kind: Instance { id: 874f6d4c-3896-4540-bb8c-21347e665438 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(12082344), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-a9ed00178c9", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 43d3e8c1-1c9d-4f8d-859f-afdc043bed22, read_only_parent: Some(Volume { block_size: 512, id: 84a74b6e-69f6-442c-bbfb-9b80084c8e3d, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 1785d715-ae6b-4e8d-87eb-34c27f37d857, key: Some("wXAwZi9nltTjbd/msPIFqjBWLowdXPFrx1mZQRJyiA4="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 43d3e8c1-1c9d-4f8d-859f-afdc043bed22, key: Some("M6UZwzKZo4nZdYMPs3lqJt+1Jk3PNYWcplgk4xlfk/Y="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::1a]:19000"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-09-23T18:27:59.443289850Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
18292024-09-23T18:27:59.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4005 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62314 req_id = c839b718-1064-449f-874f-a6ac9827e9b8 response_code = 200 uri = /vmms/e30125b6-5811-48c9-991d-b86f19bd5ffd
18302024-09-23T18:27:59.550ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44176
18312024-09-23T18:27:59.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 657 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44176 req_id = f7317f60-fb96-4b0e-99a4-a9e2c35667c3 response_code = 200 uri = /vmms/e30125b6-5811-48c9-991d-b86f19bd5ffd/state
18322024-09-23T18:27:59.553ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 12082344 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
18332024-09-23T18:27:59.553ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18342024-09-23T18:27:59.592ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd file = illumos-utils/src/zone.rs:316 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18352024-09-23T18:27:59.614ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd file = illumos-utils/src/zone.rs:354 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18362024-09-23T18:28:00.693ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18372024-09-23T18:28:00.693ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd zone file = illumos-utils/src/running_zone.rs:488 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18382024-09-23T18:28:00.873ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 62.367602ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18392024-09-23T18:28:00.960ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 117.517219ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18402024-09-23T18:28:01.098ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 295.810881ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18412024-09-23T18:28:01.409ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 586.850679ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18422024-09-23T18:28:02.011ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 731.230511ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18432024-09-23T18:28:02.119ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58603
18442024-09-23T18:28:02.181ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57663
18452024-09-23T18:28:02.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57663 req_id = 925fcaa9-24e5-44d2-9bb2-34862adc1d2d response_code = 200 uri = /vpc-routes
18462024-09-23T18:28:02.193ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61309
18472024-09-23T18:28:02.258ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:57663 req_id = 8a59f3cf-b0e7-477c-88ed-d495377a2c3a response_code = 204 uri = /vpc-routes
18482024-09-23T18:28:02.274ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59127
18492024-09-23T18:28:02.282ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55389
18502024-09-23T18:28:02.282ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18512024-09-23T18:28:02.328ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18522024-09-23T18:28:02.404ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 121790 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55389 req_id = 78629b92-5cdb-4142-9a3e-76190cb09061 response_code = 204 uri = /switch-ports
18532024-09-23T18:28:02.404ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36069
18542024-09-23T18:28:02.404ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34755
18552024-09-23T18:28:02.404ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61700
18562024-09-23T18:28:02.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 937 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36069 req_id = 28d43ab2-fb5d-48b3-87bd-01d14862a616 response_code = 200 uri = /vpc-routes
18572024-09-23T18:28:02.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 920 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34755 req_id = e20957a3-a575-422e-8559-52d44faa1d20 response_code = 200 uri = /vpc-routes
18582024-09-23T18:28:02.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1005 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61700 req_id = 99851f69-92b7-4a98-851b-5c477484f547 response_code = 200 uri = /v2p
18592024-09-23T18:28:02.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55389 req_id = 18f1f5aa-b760-46af-bb7a-7392ce9435a0 response_code = 200 uri = /network-bootstore-config
18602024-09-23T18:28:02.422ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34755 req_id = 4839edcf-720f-4261-9c63-b0f94ca392bf response_code = 204 uri = /vpc-routes
18612024-09-23T18:28:02.424ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33662
18622024-09-23T18:28:02.425ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 770 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33662 req_id = 3a79da64-bc3f-4dba-afcf-69114a81e840 response_code = 200 uri = /v2p
18632024-09-23T18:28:02.429ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36069 req_id = 1efc7279-a7bc-4253-add3-da5caaa0253d response_code = 204 uri = /vpc-routes
18642024-09-23T18:28:02.440ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52513
18652024-09-23T18:28:02.440ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18662024-09-23T18:28:02.479ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18672024-09-23T18:28:02.543ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103032 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52513 req_id = 8c6e3bcc-e9f7-4253-bcc8-030013df2bd5 response_code = 204 uri = /switch-ports
18682024-09-23T18:28:02.543ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52700
18692024-09-23T18:28:02.543ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4089
18702024-09-23T18:28:02.578ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4113
18712024-09-23T18:28:02.636ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92673 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52700 req_id = 56085aad-60ac-4f0c-88f3-15e7f4b1958c response_code = 204 uri = /switch-ports
18722024-09-23T18:28:02.636ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53635
18732024-09-23T18:28:02.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1008 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53635 req_id = 2631ed8f-59ae-48a5-b908-703fee8950c9 response_code = 200 uri = /v2p
18742024-09-23T18:28:02.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1101 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52700 req_id = 5b488039-2233-46f2-bae2-1f77e87dd5aa response_code = 200 uri = /network-bootstore-config
18752024-09-23T18:28:02.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1255 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52513 req_id = 1b323d9f-9ea0-43d9-9ac2-d0d4f3778b91 response_code = 200 uri = /network-bootstore-config
18762024-09-23T18:28:02.754ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 653.197035ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18772024-09-23T18:28:03.419ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 533.561956ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18782024-09-23T18:28:03.963ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 1.307143485s file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18792024-09-23T18:28:05.281ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 684.512633ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd zone = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
18802024-09-23T18:28:05.997ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd file = sled-agent/src/instance.rs:1466 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18812024-09-23T18:28:06.007ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 52.522628ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18822024-09-23T18:28:06.070ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 105.486386ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18832024-09-23T18:28:06.185ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd") failed: Property not found. retry in 248.936789ms file = illumos-utils/src/svc.rs:36 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18842024-09-23T18:28:06.445ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1475 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18852024-09-23T18:28:06.479ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1499 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18862024-09-23T18:28:06.619ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:744 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18872024-09-23T18:28:06.620ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4501011 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58603 req_id = 1783c882-fbdd-448d-8a8a-9184d01950fa response_code = 200 uri = /vmms/e30125b6-5811-48c9-991d-b86f19bd5ffd/state
18882024-09-23T18:28:06.620ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4426427 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61309 req_id = dea5c8ac-96b3-4559-b894-c6edb308d290 response_code = 200 uri = /vmms/e30125b6-5811-48c9-991d-b86f19bd5ffd/state
18892024-09-23T18:28:06.620ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4345748 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59127 req_id = 1e3cf673-18e3-40da-a730-ab5b275c85ed response_code = 200 uri = /vmms/e30125b6-5811-48c9-991d-b86f19bd5ffd/state
18902024-09-23T18:28:06.620ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-09-23T18:28:06.620702308Z } file = sled-agent/src/instance.rs:594 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18912024-09-23T18:28:06.620ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-23T18:28:06.620702308Z } propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18922024-09-23T18:28:06.620ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-09-23T18:28:06.620702308Z }, migration_in: None, migration_out: None }
18932024-09-23T18:28:06.671ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-09-23T18:28:06.671666825Z } file = sled-agent/src/instance.rs:594 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18942024-09-23T18:28:06.671ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-23T18:28:06.671666825Z } propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
18952024-09-23T18:28:06.671ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-09-23T18:28:06.671666825Z }, migration_in: None, migration_out: None }
18962024-09-23T18:28:07.504ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8764 local_addr = [fd00:1122:3344:101::1]:53690 method = GET remote_addr = [fd00:1122:3344:101::d]:57162 req_id = 81a4d906-70f8-40e4-bdf8-ab969f8a6c13 response_code = 200 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8 uri = /6061f953-e31a-4a2a-80c5-755d0e0127f8
18972024-09-23T18:28:12.491ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:38290
18982024-09-23T18:28:12.491ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:38290 req_id = 593833c0-dd79-4c9d-a27e-682f2045c630 response_code = 200 uri = /baseboard
18992024-09-23T18:28:17.013ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37689
19002024-09-23T18:28:17.014ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 705 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37689 req_id = cc681890-0300-4cf7-abfb-bfd0f8031815 response_code = 200 uri = /vmms/e30125b6-5811-48c9-991d-b86f19bd5ffd/state
19012024-09-23T18:28:17.014ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-09-23T18:28:17.014750541Z } file = sled-agent/src/instance.rs:594 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19022024-09-23T18:28:17.014ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-23T18:28:17.014750541Z } propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19032024-09-23T18:28:17.014ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-09-23T18:28:17.014750541Z }, migration_in: None, migration_out: None }
19042024-09-23T18:28:17.025ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-09-23T18:28:17.025548391Z } file = sled-agent/src/instance.rs:594 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19052024-09-23T18:28:17.025ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-09-23T18:28:17.025548391Z } propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19062024-09-23T18:28:17.025ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19072024-09-23T18:28:17.025ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19082024-09-23T18:28:17.025ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 11576011682804997507 n_samples = 6 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
19092024-09-23T18:28:17.026ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 6467436795131919291 n_samples = 6 sled_id = 6061f953-e31a-4a2a-80c5-755d0e0127f8
19102024-09-23T18:28:17.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19112024-09-23T18:28:17.028ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/a6d75f0e-db4f-4052-94fb-1f0675a27263/debug/bundle/zone", "/pool/int/f181c9e6-1009-477f-8ae5-05396e960261/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/016c0356-f4b4-4d6b-8c24-edfd2951cfe3/crypt/debug", "/pool/ext/b5c5f1b9-c752-4261-a49d-06cee1ff1f1a/crypt/debug", "/pool/ext/e381d5a5-91f1-4b1c-b0ad-128e813a0a94/crypt/debug", "/pool/ext/67a20504-7be3-48ff-bb32-7f9411b1342b/crypt/debug", "/pool/ext/4edf146e-3ed8-4c23-b131-d4ed33864007/crypt/debug", "/pool/ext/17d503a6-b44e-42a3-ae74-5b19be755ad5/crypt/debug", "/pool/ext/1f7ede35-40db-4a1a-8087-fc87b6705700/crypt/debug", "/pool/ext/c6321c72-f89d-471a-8986-1f1bf65a3de2/crypt/debug", "/pool/ext/73f13013-70b2-4020-853f-5a70920c296f/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd
19122024-09-23T18:28:17.628ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd", bundle_id: 61a17aa8-e64c-4da8-b936-0bfd16e2cd11 }, time_created: 2024-09-23T18:28:17.028189967Z, version: 0, cause: TerminatedInstance }
19132024-09-23T18:28:17.628ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_e30125b6-5811-48c9-991d-b86f19bd5ffd file = sled-agent/src/instance.rs:843 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19142024-09-23T18:28:19.354ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19152024-09-23T18:28:19.370ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(12082344), kind: System }
19162024-09-23T18:28:19.371ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(12082344), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
19172024-09-23T18:28:19.371ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-23T18:28:19.371349281Z }, migration_in: None, migration_out: None }
19182024-09-23T18:28:19.421ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-09-23T18:28:19.371349281Z }, migration_in: None, migration_out: None }
19192024-09-23T18:28:19.425ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:780 instance_id = 874f6d4c-3896-4540-bb8c-21347e665438 propolis_id = e30125b6-5811-48c9-991d-b86f19bd5ffd
19202024-09-23T18:28:19.912ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38552
19212024-09-23T18:28:19.912ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38552 req_id = 89f37778-6b4a-4d2d-a933-2cd31b1390da response_code = 200 uri = /vpc-routes
19222024-09-23T18:28:19.922ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38552 req_id = ba3277ed-39fd-41bf-96fe-40ef6e22560d response_code = 204 uri = /vpc-routes
19232024-09-23T18:28:20.014ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47251
19242024-09-23T18:28:20.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 640 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47251 req_id = e54179a2-6e6b-4477-b22d-c4d3a2c7de5e response_code = 200 uri = /v2p
19252024-09-23T18:28:20.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40995
19262024-09-23T18:28:20.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40995 req_id = 4e28df08-a60b-46b8-9365-a7d0a65dc612 response_code = 200 uri = /v2p
19272024-09-23T18:28:20.745ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39339
19282024-09-23T18:28:20.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39339 req_id = 5467cd29-e218-4b4c-9385-f1078a1f3e9e response_code = 200 uri = /vpc-routes
19292024-09-23T18:28:20.753ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 682 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39339 req_id = 7da0b97a-0980-4b99-8a6c-9096f0d2f337 response_code = 204 uri = /vpc-routes