Max level shown:
1 [ Oct 17 15:04:14 Enabled. ]
2 [ Oct 17 15:04:14 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 17 15:04:14 Method "start" exited with status 0. ]
4 [ Oct 17 15:04:14 Rereading configuration. ]
5 [ Oct 17 15:04:14 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-17T15:04:14.404ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-17T15:04:14.404ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-17T15:04:14.457ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-17T15:04:14.457ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-17T15:04:14.570ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-17T15:04:14.578ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-17T15:04:14.587ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-17T15:04:15.188ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:9fb2::, width: 64 }
152024-10-17T15:04:15.323ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-17T15:04:15.323ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-17T15:04:15.323ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-17T15:04:15.323ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-10-17T15:04:15.323ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-10-17T15:04:15.323ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-17T15:04:15.326ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "buskin", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-17T15:04:15.326ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-17T15:04:15.326ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-17T15:04:15.326ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-17T15:04:15.326ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-17T15:04:15.326ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-17T15:04:15.326ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-17T15:04:15.326ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-17T15:04:15.326ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-17T15:04:15.338ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-17T15:04:15.338ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 34423522-8473-4e67-b21d-dccdbba2931c (zpool)
322024-10-17T15:04:15.455ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal }
332024-10-17T15:04:15.934ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-17T15:04:15.934ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-17T15:04:15.934ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-17T15:04:15.934ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-17T15:04:15.934ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-10-17T15:04:15.934ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-10-17T15:04:15.934ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-10-17T15:04:15.942ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-10-17T15:04:15.942ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool)
422024-10-17T15:04:16.051ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal }
432024-10-17T15:04:16.530ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-10-17T15:04:16.530ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-10-17T15:04:16.530ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-10-17T15:04:16.530ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-10-17T15:04:16.530ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-10-17T15:04:16.530ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-10-17T15:04:16.530ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
522024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
532024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
542024-10-17T15:04:16.530ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-10-17T15:04:16.530ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-10-17T15:04:16.530ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
572024-10-17T15:04:16.530ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
582024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
592024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
602024-10-17T15:04:16.530ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-10-17T15:04:16.530ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-10-17T15:04:16.530ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-10-17T15:04:16.530ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-10-17T15:04:16.530ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-10-17T15:04:16.530ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-10-17T15:04:16.530ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-10-17T15:04:16.530ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
702024-10-17T15:04:16.530ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
712024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
722024-10-17T15:04:16.530ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
732024-10-17T15:04:16.530ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-10-17T15:04:16.530ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-10-17T15:04:16.531ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-10-17T15:04:16.531ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-10-17T15:04:16.531ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-10-17T15:04:16.531ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
812024-10-17T15:04:16.531ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
822024-10-17T15:04:16.531ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-10-17T15:04:16.531ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-10-17T15:04:16.531ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-10-17T15:04:16.531ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
882024-10-17T15:04:16.531ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
892024-10-17T15:04:16.531ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
902024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
912024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
922024-10-17T15:04:16.531ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-10-17T15:04:16.531ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-10-17T15:04:16.531ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
952024-10-17T15:04:16.531ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
962024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
972024-10-17T15:04:16.531ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
982024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
992024-10-17T15:04:16.531ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1002024-10-17T15:04:16.531ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-10-17T15:04:16.531ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-10-17T15:04:16.531ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-10-17T15:04:16.531ZINFOSledAgent (bootstore): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1052024-10-17T15:04:16.531ZINFOSledAgent (bootstore): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1062024-10-17T15:04:16.531ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-buskin-i86pc
1072024-10-17T15:04:16.531ZINFOSledAgent (bootstore): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1082024-10-17T15:04:16.531ZINFOSledAgent (bootstore): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-buskin-i86pc
1092024-10-17T15:04:16.531ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-buskin-i86pc
1102024-10-17T15:04:16.531ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-10-17T15:04:16.531ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-10-17T15:04:16.542ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:739
1132024-10-17T15:04:16.542ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 34489, tv_nsec: 500002363 } time_to_next_cleanup = 599.999922907s
1142024-10-17T15:04:16.542ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1152024-10-17T15:04:16.542ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3924
1162024-10-17T15:04:16.542ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-10-17T15:04:16.545ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-10-17T15:04:16.545ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80
1192024-10-17T15:04:16.547ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:12346
1202024-10-17T15:04:16.547ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1073
1212024-10-17T15:04:16.547ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1222024-10-17T15:04:16.547ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-10-17T15:04:16.547ZINFOSledAgent (RSS): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-10-17T15:04:16.547ZINFOSledAgent (RSS): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-10-17T15:04:16.547ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1136
1262024-10-17T15:04:16.547ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-10-17T15:04:16.547ZINFOSledAgent (RSS): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-10-17T15:04:16.547ZINFOSledAgent (RSS): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-10-17T15:04:16.547ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1178
1302024-10-17T15:04:16.547ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-10-17T15:04:16.547ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:9fb2::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-10-17T15:04:16.547ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-10-17T15:04:16.547ZINFOSledAgent (RSS): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-10-17T15:04:16.548ZINFOSledAgent (RSS): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-10-17T15:04:16.548ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: b4d984d4-c87e-4329-9460-121af5c9b7fe, sleds: { [fdb0:18c0:4d0d:9fb2::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 (sled), rack_id: b4d984d4-c87e-4329-9460-121af5c9b7fe, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.1.164, last: 10.151.1.173, }, ), ], external_dns_ips: [ 10.151.1.164, 10.151.1.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-10-17T15:04:16.548ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1212
1372024-10-17T15:04:16.548ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
1382024-10-17T15:04:16.548ZINFOSledAgent (bootstore): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1392024-10-17T15:04:16.548ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
1402024-10-17T15:04:16.548ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
1412024-10-17T15:04:16.548ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 (sled), rack_id: b4d984d4-c87e-4329-9460-121af5c9b7fe, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
1422024-10-17T15:04:16.549ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4221
1432024-10-17T15:04:16.549ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4603
1442024-10-17T15:04:16.554ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-10-17T15:04:16.560ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-10-17T15:04:16.560ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:9fb2::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-10-17T15:04:16.564ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-10-17T15:04:16.564ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-10-17T15:04:16.564ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:59827
1502024-10-17T15:04:16.564ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 (sled), rack_id: b4d984d4-c87e-4329-9460-121af5c9b7fe, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1512024-10-17T15:04:16.564ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1522024-10-17T15:04:16.565ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-10-17T15:04:16.565ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1542024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1552024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1562024-10-17T15:04:16.565ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1572024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1582024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-10-17T15:04:16.565ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-10-17T15:04:16.565ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-10-17T15:04:16.580ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-10-17T15:04:16.600ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-10-17T15:04:16.618ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-10-17T15:04:16.629ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
1682024-10-17T15:04:16.629ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-10-17T15:04:16.629ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
1702024-10-17T15:04:16.629ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1712024-10-17T15:04:16.632ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 62.127022ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fb51405f-4d4b-4a96-ac61-07ec6fa0ef56", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fb51405f-4d4b-4a96-ac61-07ec6fa0ef56" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-10-17T15:04:16.643ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_34423522-8473-4e67-b21d-dccdbba2931c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-10-17T15:04:16.673ZINFOSledAgent: successfully created encrypted zvol: name="oxi_34423522-8473-4e67-b21d-dccdbba2931c/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-10-17T15:04:16.673ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_34423522-8473-4e67-b21d-dccdbba2931c/swap" file = sled-agent/src/swap_device.rs:96
1752024-10-17T15:04:16.678ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
1762024-10-17T15:04:16.678ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-10-17T15:04:16.678ZINFOSledAgent (BackingFs): Ensuring dataset oxi_34423522-8473-4e67-b21d-dccdbba2931c/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-10-17T15:04:16.695ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-10-17T15:04:16.696ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 138.929343ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "94945585-3d0f-4142-899c-98ab7ed283d4", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "94945585-3d0f-4142-899c-98ab7ed283d4" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-10-17T15:04:16.783ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1812024-10-17T15:04:16.835ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-17T15:04:16.836ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 196.507599ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ce306f68-0970-43f9-994d-ae9601e465ee", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ce306f68-0970-43f9-994d-ae9601e465ee" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-17T15:04:17.003ZINFOSledAgent (BackingFs): Mounting oxi_34423522-8473-4e67-b21d-dccdbba2931c/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1842024-10-17T15:04:17.018ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1852024-10-17T15:04:17.018ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1862024-10-17T15:04:17.018ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1872024-10-17T15:04:17.018ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1882024-10-17T15:04:17.033ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-10-17T15:04:17.034ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 122.100029ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a8e22ab3-0865-4eef-a651-232132df9cd7", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a8e22ab3-0865-4eef-a651-232132df9cd7" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-10-17T15:04:17.157ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-10-17T15:04:17.158ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 208.695586ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4eb5eea5-bb24-4cdf-aaa6-918ed831279c", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4eb5eea5-bb24-4cdf-aaa6-918ed831279c" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-10-17T15:04:17.368ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-10-17T15:04:17.368ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 184.807395ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "86201118-aa17-4d01-9800-f2db73afe02b", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "86201118-aa17-4d01-9800-f2db73afe02b" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-10-17T15:04:17.555ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-10-17T15:04:17.555ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 342.788808ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "71ce867e-ad52-4f44-b6ed-bd7d42a021df", "content-length": "124", "date": "Thu, 17 Oct 2024 15:04:16 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "71ce867e-ad52-4f44-b6ed-bd7d42a021df" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-10-17T15:04:17.756ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
1972024-10-17T15:04:17.762ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:44276 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
1982024-10-17T15:04:17.762ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
1992024-10-17T15:04:17.762ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:44276 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2002024-10-17T15:04:17.763ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 356.548411ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2012024-10-17T15:04:17.764ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2022024-10-17T15:04:17.764ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2032024-10-17T15:04:17.764ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:923 underlay_address = fd00:1122:3344:101::1
2042024-10-17T15:04:17.764ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2052024-10-17T15:04:17.900ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2062024-10-17T15:04:18.121ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 676.755475ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2072024-10-17T15:04:18.800ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 624.582868ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2082024-10-17T15:04:19.426ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.103043197s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2092024-10-17T15:04:20.163ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2679
2102024-10-17T15:04:20.163ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2897
2112024-10-17T15:04:20.163ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2549
2122024-10-17T15:04:20.163ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2596
2132024-10-17T15:04:20.163ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2978
2142024-10-17T15:04:20.163ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3020
2152024-10-17T15:04:20.165ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2822
2162024-10-17T15:04:20.166ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2590
2172024-10-17T15:04:20.167ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='8d07ef28-1987-4453-855f-1cd688dbea37'/> <propval type="astring" name="rack_id" value='b4d984d4-c87e-4329-9460-121af5c9b7fe'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:9fb2::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:feeb:b7a'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:9fb2::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"buskin","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4'/> <propval type="astring" name="rack_id" value='b4d984d4-c87e-4329-9460-121af5c9b7fe'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='buskin'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4'/> <propval type="astring" name="rack_uuid" value='b4d984d4-c87e-4329-9460-121af5c9b7fe'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4'/> <propval type="astring" name="rack_uuid" value='b4d984d4-c87e-4329-9460-121af5c9b7fe'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2182024-10-17T15:04:20.167ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2192024-10-17T15:04:20.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 40.197947ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-10-17T15:04:20.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 115.358379ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-17T15:04:20.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 132.501042ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-10-17T15:04:21.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 371.748415ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-17T15:04:21.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 795.29818ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-10-17T15:04:21.716ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.933002784s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2252024-10-17T15:04:22.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 913.389578ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-10-17T15:04:23.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.172931525s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-17T15:04:24.652ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 11.859361709s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2282024-10-17T15:04:25.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.01143225s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2292024-10-17T15:04:26.572ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2302024-10-17T15:04:26.572ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2312024-10-17T15:04:26.572ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2322024-10-17T15:04:26.572ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-10-17T15:04:26.572ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2342024-10-17T15:04:26.572ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2352024-10-17T15:04:26.572ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.1.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.1.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2362024-10-17T15:04:26.572ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3924
2372024-10-17T15:04:26.572ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:9fb2::1]:59827
2382024-10-17T15:04:26.783ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:9fb2::1]:12346
2392024-10-17T15:04:26.783ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:9fb2::1]:12346
2402024-10-17T15:04:26.783ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2412024-10-17T15:04:26.783ZINFOSledAgent (RSS): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-10-17T15:04:26.783ZINFOSledAgent (RSS): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-10-17T15:04:26.783ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-10-17T15:04:26.783ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-17T15:04:26.783ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-10-17T15:04:26.788ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4245 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2472024-10-17T15:04:26.788ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4261
2482024-10-17T15:04:26.788ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2492024-10-17T15:04:27.074ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43993
2502024-10-17T15:04:27.075ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2512024-10-17T15:04:27.075ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2522024-10-17T15:04:27.075ZINFOSledAgent (ServiceManager): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:188
2532024-10-17T15:04:27.075ZINFOSledAgent (ServiceManager): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:188
2542024-10-17T15:04:27.075ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 335 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43993 req_id = 7f4dec6b-769d-4320-b4a3-75d26664df7f response_code = 200 uri = /inventory
2552024-10-17T15:04:27.105ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44482
2562024-10-17T15:04:27.105ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44482 req_id = ba5c2690-a178-4461-bec3-c02ff8bd5088 response_code = 200 uri = /sled-role
2572024-10-17T15:04:27.106ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2582024-10-17T15:04:27.106ZINFOSledAgent (RSS): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2592024-10-17T15:04:27.107ZINFOSledAgent (RSS): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2602024-10-17T15:04:27.107ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:882
2612024-10-17T15:04:27.139ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: ff79f81f-54e7-4b44-be68-aaab5a7a9b97, pool_id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 6a51afd0-c394-4bab-85eb-70a53755be1d, pool_id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 21323c8f-423c-4bc4-89eb-5d8d8061d33a, pool_id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 20aee0af-a94f-46fb-af59-edcd6b387347, pool_id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 63ac48bf-b682-484d-b53d-9eece22cdde1, pool_id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 944cf4fa-850a-4ec3-a588-4117ec256289, pool_id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 4ce73b27-fcb3-45e6-b665-cceb1b36165f, pool_id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 67216b71-9df1-485f-9e47-60ea6c8db045, pool_id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: a3da30da-c02c-46aa-bc4b-880a6f1be473, pool_id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2622024-10-17T15:04:27.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59718
2632024-10-17T15:04:27.139ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2642024-10-17T15:04:27.139ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: ff79f81f-54e7-4b44-be68-aaab5a7a9b97, pool_id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 6a51afd0-c394-4bab-85eb-70a53755be1d, pool_id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 21323c8f-423c-4bc4-89eb-5d8d8061d33a, pool_id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 20aee0af-a94f-46fb-af59-edcd6b387347, pool_id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 63ac48bf-b682-484d-b53d-9eece22cdde1, pool_id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 944cf4fa-850a-4ec3-a588-4117ec256289, pool_id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 4ce73b27-fcb3-45e6-b665-cceb1b36165f, pool_id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 67216b71-9df1-485f-9e47-60ea6c8db045, pool_id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: a3da30da-c02c-46aa-bc4b-880a6f1be473, pool_id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2652024-10-17T15:04:27.139ZINFOSledAgent (StorageManager): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2662024-10-17T15:04:27.139ZINFOSledAgent (StorageManager): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2672024-10-17T15:04:27.139ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2682024-10-17T15:04:27.139ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2692024-10-17T15:04:27.139ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2702024-10-17T15:04:27.139ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2712024-10-17T15:04:27.139ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2722024-10-17T15:04:27.267ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2732024-10-17T15:04:27.267ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool)
2742024-10-17T15:04:27.716ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/probe_manager.rs:230 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2752024-10-17T15:04:27.934ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2762024-10-17T15:04:27.935ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2772024-10-17T15:04:27.935ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
2782024-10-17T15:04:27.993ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }
2792024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2802024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2812024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2822024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2832024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2842024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2852024-10-17T15:04:28.000ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2862024-10-17T15:04:28.042ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2872024-10-17T15:04:28.050ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_86d90d1a-236b-48f7-9d13-e0e08c6f416e/crypt/zone file = sled-storage/src/dataset.rs:280
2882024-10-17T15:04:28.252ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_86d90d1a-236b-48f7-9d13-e0e08c6f416e file = sled-storage/src/dataset.rs:366
2892024-10-17T15:04:28.259ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2902024-10-17T15:04:28.259ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2912024-10-17T15:04:28.259ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2922024-10-17T15:04:28.259ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2932024-10-17T15:04:28.259ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2942024-10-17T15:04:28.269ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2952024-10-17T15:04:28.269ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool)
2962024-10-17T15:04:28.399ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }
2972024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2982024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
2992024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
3002024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
3012024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3022024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3032024-10-17T15:04:28.407ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3042024-10-17T15:04:28.447ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3052024-10-17T15:04:28.455ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6bbd0bbe-4092-4d91-b621-d8484bd8935a/crypt/zone file = sled-storage/src/dataset.rs:280
3062024-10-17T15:04:28.660ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6bbd0bbe-4092-4d91-b621-d8484bd8935a file = sled-storage/src/dataset.rs:366
3072024-10-17T15:04:28.668ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3082024-10-17T15:04:28.668ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3092024-10-17T15:04:28.668ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3102024-10-17T15:04:28.668ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3112024-10-17T15:04:28.668ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3122024-10-17T15:04:28.678ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3132024-10-17T15:04:28.678ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool)
3142024-10-17T15:04:28.789ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }
3152024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3162024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3172024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3182024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3192024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3202024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3212024-10-17T15:04:28.797ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3222024-10-17T15:04:28.800ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/probe_manager.rs:230 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
3232024-10-17T15:04:28.841ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3242024-10-17T15:04:28.849ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_13507129-93b4-4cdb-aa3e-5a74ab4d7515/crypt/zone file = sled-storage/src/dataset.rs:280
3252024-10-17T15:04:29.055ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_13507129-93b4-4cdb-aa3e-5a74ab4d7515 file = sled-storage/src/dataset.rs:366
3262024-10-17T15:04:29.063ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3272024-10-17T15:04:29.063ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3282024-10-17T15:04:29.063ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3292024-10-17T15:04:29.063ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3302024-10-17T15:04:29.063ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3312024-10-17T15:04:29.072ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3322024-10-17T15:04:29.072ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool)
3332024-10-17T15:04:29.192ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }
3342024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3352024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3362024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3372024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3382024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3392024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3402024-10-17T15:04:29.199ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3412024-10-17T15:04:29.241ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3422024-10-17T15:04:29.248ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0/crypt/zone file = sled-storage/src/dataset.rs:280
3432024-10-17T15:04:29.450ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 file = sled-storage/src/dataset.rs:366
3442024-10-17T15:04:29.458ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3452024-10-17T15:04:29.458ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3462024-10-17T15:04:29.458ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3472024-10-17T15:04:29.458ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3482024-10-17T15:04:29.458ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3492024-10-17T15:04:29.467ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3502024-10-17T15:04:29.467ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool)
3512024-10-17T15:04:29.580ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }
3522024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3532024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3542024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3552024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3562024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3572024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3582024-10-17T15:04:29.589ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3592024-10-17T15:04:29.626ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3602024-10-17T15:04:29.634ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f/crypt/zone file = sled-storage/src/dataset.rs:280
3612024-10-17T15:04:29.803ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/probe_manager.rs:230 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
3622024-10-17T15:04:29.840ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f file = sled-storage/src/dataset.rs:366
3632024-10-17T15:04:29.848ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3642024-10-17T15:04:29.848ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3652024-10-17T15:04:29.848ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3662024-10-17T15:04:29.848ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3672024-10-17T15:04:29.848ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3682024-10-17T15:04:29.859ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3692024-10-17T15:04:29.859ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool)
3702024-10-17T15:04:29.978ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }
3712024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3722024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3732024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3742024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3752024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3762024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3772024-10-17T15:04:29.987ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3782024-10-17T15:04:30.034ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3792024-10-17T15:04:30.042ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e5d462a9-5343-4c50-9dc4-2adbb2b66d5e/crypt/zone file = sled-storage/src/dataset.rs:280
3802024-10-17T15:04:30.253ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e5d462a9-5343-4c50-9dc4-2adbb2b66d5e file = sled-storage/src/dataset.rs:366
3812024-10-17T15:04:30.261ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3822024-10-17T15:04:30.261ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3832024-10-17T15:04:30.261ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3842024-10-17T15:04:30.261ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3852024-10-17T15:04:30.261ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3862024-10-17T15:04:30.271ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3872024-10-17T15:04:30.271ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool)
3882024-10-17T15:04:30.411ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool), kind: External }
3892024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3902024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3912024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3922024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3932024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3942024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3952024-10-17T15:04:30.419ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3962024-10-17T15:04:30.466ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3972024-10-17T15:04:30.475ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf/crypt/zone file = sled-storage/src/dataset.rs:280
3982024-10-17T15:04:30.678ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf file = sled-storage/src/dataset.rs:366
3992024-10-17T15:04:30.686ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
4002024-10-17T15:04:30.686ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4012024-10-17T15:04:30.686ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4022024-10-17T15:04:30.686ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4032024-10-17T15:04:30.686ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4042024-10-17T15:04:30.697ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4052024-10-17T15:04:30.697ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 6dc43632-8304-473f-ba75-413bbd171b4c (zpool)
4062024-10-17T15:04:30.818ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool), kind: External }
4072024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4082024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4092024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4102024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4112024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4122024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4132024-10-17T15:04:30.825ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4142024-10-17T15:04:30.829ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/probe_manager.rs:230 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4152024-10-17T15:04:30.870ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4162024-10-17T15:04:30.878ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6dc43632-8304-473f-ba75-413bbd171b4c/crypt/zone file = sled-storage/src/dataset.rs:280
4172024-10-17T15:04:31.084ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6dc43632-8304-473f-ba75-413bbd171b4c file = sled-storage/src/dataset.rs:366
4182024-10-17T15:04:31.092ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4192024-10-17T15:04:31.092ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4202024-10-17T15:04:31.093ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4212024-10-17T15:04:31.093ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4222024-10-17T15:04:31.093ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4232024-10-17T15:04:31.102ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4242024-10-17T15:04:31.102ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool)
4252024-10-17T15:04:31.214ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External }
4262024-10-17T15:04:31.221ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4272024-10-17T15:04:31.222ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4282024-10-17T15:04:31.222ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4292024-10-17T15:04:31.222ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4302024-10-17T15:04:31.222ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4312024-10-17T15:04:31.222ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4322024-10-17T15:04:31.222ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4332024-10-17T15:04:31.263ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4342024-10-17T15:04:31.271ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6931ef6e-c78c-4d61-ab93-92f35309d55f/crypt/zone file = sled-storage/src/dataset.rs:280
4352024-10-17T15:04:31.479ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6931ef6e-c78c-4d61-ab93-92f35309d55f file = sled-storage/src/dataset.rs:366
4362024-10-17T15:04:31.487ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4372024-10-17T15:04:31.487ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4382024-10-17T15:04:31.487ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4392024-10-17T15:04:31.488ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 34423522-8473-4e67-b21d-dccdbba2931c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2af104c1-ee2c-49fb-8805-3ed22aa08cdd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4402024-10-17T15:04:31.488ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4412024-10-17T15:04:31.488ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4422024-10-17T15:04:31.488ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4432024-10-17T15:04:31.488ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4442024-10-17T15:04:31.488ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4452024-10-17T15:04:31.488ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4462024-10-17T15:04:31.488ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4472024-10-17T15:04:31.488ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4482024-10-17T15:04:31.488ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4492024-10-17T15:04:31.488ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4502024-10-17T15:04:31.488ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:798 gen = Generation(1)
4512024-10-17T15:04:31.488ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4522024-10-17T15:04:31.488ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4349268 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59718 req_id = 7804f697-b8f7-496f-8c98-dd8d60f59b63 response_code = 200 uri = /omicron-physical-disks
4532024-10-17T15:04:31.519ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 764d7304-941c-47d5-bd71-20d679249cf8 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 703a01d4-fb0e-4834-936c-a729fd81d47b (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4542024-10-17T15:04:31.519ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53413
4552024-10-17T15:04:31.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4562024-10-17T15:04:31.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }
4572024-10-17T15:04:31.579ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4582024-10-17T15:04:31.579ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }
4592024-10-17T15:04:31.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4602024-10-17T15:04:31.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4612024-10-17T15:04:31.696ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4622024-10-17T15:04:31.696ZINFOSledAgent (ServiceManager): No ledger in /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:188
4632024-10-17T15:04:31.696ZINFOSledAgent (ServiceManager): No ledger in /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:188
4642024-10-17T15:04:31.696ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-10-17T15:04:31.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4662024-10-17T15:04:31.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4672024-10-17T15:04:31.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4682024-10-17T15:04:31.791ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8 file = illumos-utils/src/zone.rs:316
4692024-10-17T15:04:31.795ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b file = illumos-utils/src/zone.rs:316
4702024-10-17T15:04:31.799ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f file = illumos-utils/src/zone.rs:316
4712024-10-17T15:04:31.812ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8 file = illumos-utils/src/zone.rs:354
4722024-10-17T15:04:31.816ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b file = illumos-utils/src/zone.rs:354
4732024-10-17T15:04:31.820ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f file = illumos-utils/src/zone.rs:354
4742024-10-17T15:04:31.832ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/probe_manager.rs:230 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4752024-10-17T15:04:32.834ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/probe_manager.rs:230 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
4762024-10-17T15:04:33.419ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4772024-10-17T15:04:33.420ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4782024-10-17T15:04:33.420ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
4792024-10-17T15:04:33.904ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4802024-10-17T15:04:33.904ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4812024-10-17T15:04:34.383ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4822024-10-17T15:04:34.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 27.65646ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
4832024-10-17T15:04:34.390ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
4842024-10-17T15:04:34.394ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4852024-10-17T15:04:34.394ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
4862024-10-17T15:04:34.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 81.833638ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
4872024-10-17T15:04:34.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 28.294067ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
4882024-10-17T15:04:34.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 184.213082ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
4892024-10-17T15:04:34.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 68.790179ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
4902024-10-17T15:04:34.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 121.782547ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
4912024-10-17T15:04:34.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 128.830841ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
4922024-10-17T15:04:34.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 273.592549ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
4932024-10-17T15:04:34.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 367.95565ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
4942024-10-17T15:04:34.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 102.166628ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
4952024-10-17T15:04:34.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 262.23801ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
4962024-10-17T15:04:34.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 558.824803ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
4972024-10-17T15:04:35.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 619.821012ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
4982024-10-17T15:04:35.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 473.572621ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
4992024-10-17T15:04:35.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 881.117905ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
5002024-10-17T15:04:35.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 1.148604237s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
5012024-10-17T15:04:35.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 1.337004505s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
5022024-10-17T15:04:36.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 594.272473ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
5032024-10-17T15:04:36.514ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4269
5042024-10-17T15:04:36.514ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4279 gateway = fd00:1122:3344:101::1
5052024-10-17T15:04:36.522ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4358
5062024-10-17T15:04:36.575ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7105636974374909681 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
5072024-10-17T15:04:36.575ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10045244912119534615 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
5082024-10-17T15:04:36.642ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4406
5092024-10-17T15:04:36.643ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4433
5102024-10-17T15:04:36.688ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4446
5112024-10-17T15:04:36.688ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4310
5122024-10-17T15:04:36.746ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4352
5132024-10-17T15:04:36.746ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4413 rack_subnet = fd00:1122:3344::
5142024-10-17T15:04:36.770ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4424
5152024-10-17T15:04:36.770ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4466
5162024-10-17T15:04:36.844ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4498
5172024-10-17T15:04:36.844ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4504
5182024-10-17T15:04:36.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 1.380168937s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
5192024-10-17T15:04:36.936ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4545
5202024-10-17T15:04:36.936ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
5212024-10-17T15:04:36.968ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4) file = sled-agent/src/nexus.rs:527 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
5222024-10-17T15:04:36.968ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 37.074595ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5232024-10-17T15:04:37.006ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 112.144934ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5242024-10-17T15:04:37.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 1.376048023s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
5252024-10-17T15:04:37.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 1.138317949s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
5262024-10-17T15:04:37.119ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 289.540546ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5272024-10-17T15:04:37.412ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:418
5282024-10-17T15:04:37.420ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:44872
5292024-10-17T15:04:37.420ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:44872 req_id = 90818e0e-2afa-4739-836b-336ad86c054d response_code = 200 uri = /sled-identifiers
5302024-10-17T15:04:37.445ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:686
5312024-10-17T15:04:37.445ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:724
5322024-10-17T15:04:37.445ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:727 uptime = 0
5332024-10-17T15:04:37.445ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.1.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:440
5342024-10-17T15:04:37.483ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
5352024-10-17T15:04:37.531ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5362024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5372024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5382024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5392024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5402024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5412024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5422024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5432024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5442024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5452024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5462024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5472024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5482024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5492024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5502024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5512024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5522024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5532024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5542024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5552024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5562024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5572024-10-17T15:04:37.532ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5582024-10-17T15:04:37.938ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
5592024-10-17T15:04:37.938ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
5602024-10-17T15:04:38.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b") failed: Property not found. retry in 1.397599177s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
5612024-10-17T15:04:38.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 916.165506ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
5622024-10-17T15:04:38.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 1.067036729s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
5632024-10-17T15:04:39.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f") failed: Property not found. retry in 944.419558ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
5642024-10-17T15:04:39.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8") failed: Property not found. retry in 883.429935ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
5652024-10-17T15:04:40.415ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_764d7304-941c-47d5-bd71-20d679249cf8
5662024-10-17T15:04:40.415ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_703a01d4-fb0e-4834-936c-a729fd81d47b
5672024-10-17T15:04:40.415ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_96c809da-b476-4ddd-bbdc-51baf4d3e56f
5682024-10-17T15:04:40.415ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5692024-10-17T15:04:40.415ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5702024-10-17T15:04:40.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8895864 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53413 req_id = 38c3c4ff-5c24-4073-995e-0165ec2c5592 response_code = 204 uri = /omicron-zones
5712024-10-17T15:04:40.417ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5722024-10-17T15:04:40.473ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5732024-10-17T15:04:40.531ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5742024-10-17T15:04:40.594ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5752024-10-17T15:04:40.594ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5762024-10-17T15:04:40.626ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5772024-10-17T15:04:40.627ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5782024-10-17T15:04:40.627ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5792024-10-17T15:04:40.658ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 764d7304-941c-47d5-bd71-20d679249cf8 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 703a01d4-fb0e-4834-936c-a729fd81d47b (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 24c32b69-efe8-479e-ba18-19f8c4fd316f (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: c7a31b26-8275-490c-9a47-724548eb0d53, kind: Service { id: 24c32b69-efe8-479e-ba18-19f8c4fd316f }, name: Name("ntp-24c32b69-efe8-479e-ba18-19f8c4fd316f"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 52])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5802024-10-17T15:04:40.658ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40600
5812024-10-17T15:04:40.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5822024-10-17T15:04:40.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }
5832024-10-17T15:04:40.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5842024-10-17T15:04:40.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }
5852024-10-17T15:04:40.694ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5862024-10-17T15:04:40.694ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5872024-10-17T15:04:40.712ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5882024-10-17T15:04:40.712ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:177
5892024-10-17T15:04:40.712ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:177
5902024-10-17T15:04:40.712ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5912024-10-17T15:04:40.723ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5922024-10-17T15:04:40.755ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5932024-10-17T15:04:40.756ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5942024-10-17T15:04:40.756ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5952024-10-17T15:04:40.785ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 132, 52]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5962024-10-17T15:04:40.785ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = boundary_ntp
5972024-10-17T15:04:40.786ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:83
5982024-10-17T15:04:40.786ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:90
5992024-10-17T15:04:40.786ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.169 file = clients/dpd-client/src/lib.rs:118
6002024-10-17T15:04:40.787ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6012024-10-17T15:04:40.812ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f file = illumos-utils/src/zone.rs:316
6022024-10-17T15:04:40.831ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f file = illumos-utils/src/zone.rs:354
6032024-10-17T15:04:41.466ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6042024-10-17T15:04:41.466ZINFOSledAgent (ServiceManager): Booting oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6052024-10-17T15:04:41.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 72.403336ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6062024-10-17T15:04:41.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 59.965589ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6072024-10-17T15:04:41.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 289.563359ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6082024-10-17T15:04:42.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 417.065331ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6092024-10-17T15:04:42.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 872.802839ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6102024-10-17T15:04:43.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 630.39079ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6112024-10-17T15:04:44.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 921.152014ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6122024-10-17T15:04:44.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 1.442808606s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6132024-10-17T15:04:46.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f") failed: Property not found. retry in 1.223556565s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6142024-10-17T15:04:46.577ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7105636974374909681 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6152024-10-17T15:04:46.577ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10045244912119534615 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6162024-10-17T15:04:47.640ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f
6172024-10-17T15:04:47.640ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6182024-10-17T15:04:47.640ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6192024-10-17T15:04:47.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6982439 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:40600 req_id = d283187d-116c-4218-85f0-a49c6ead8a25 response_code = 204 uri = /omicron-zones
6202024-10-17T15:04:47.643ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6212024-10-17T15:04:47.672ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6222024-10-17T15:04:47.672ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42603
6232024-10-17T15:04:47.682ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6242024-10-17T15:04:47.682ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 9410 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42603 req_id = b8275082-401b-4f12-8a59-ac70e9e34c33 response_code = 500 uri = /timesync
6252024-10-17T15:04:47.685ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6262024-10-17T15:04:47.779ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6272024-10-17T15:04:47.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55882
6282024-10-17T15:04:47.939ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6292024-10-17T15:04:47.939ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6302024-10-17T15:04:49.642ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1315565006152006779 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6312024-10-17T15:04:50.153ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11340141748898181460 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6322024-10-17T15:04:50.418ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6540327970462295184 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6332024-10-17T15:04:51.788ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6342024-10-17T15:04:51.788ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008614 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55882 req_id = 5231ad44-c6c9-4d6b-91e0-7269fcd1275f response_code = 500 uri = /timesync
6352024-10-17T15:04:51.790ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6362024-10-17T15:04:51.897ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6372024-10-17T15:04:51.897ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45515
6382024-10-17T15:04:51.905ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6392024-10-17T15:04:51.905ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8066 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45515 req_id = 100a39bb-dbe7-4326-9204-74aa562c75b8 response_code = 500 uri = /timesync
6402024-10-17T15:04:51.907ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6412024-10-17T15:04:52.051ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6422024-10-17T15:04:52.051ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41115
6432024-10-17T15:04:56.072ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_24c32b69-efe8-479e-ba18-19f8c4fd316f': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6442024-10-17T15:04:56.072ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008917 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41115 req_id = 3ddf3dfb-d29e-4648-8b6a-5c342ec8e9c1 response_code = 500 uri = /timesync
6452024-10-17T15:04:56.075ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6462024-10-17T15:04:56.346ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6472024-10-17T15:04:56.346ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54914
6482024-10-17T15:04:56.355ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8448 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54914 req_id = 07ca2a0a-e080-479d-8b1a-21fadbdb1ab9 response_code = 200 uri = /timesync
6492024-10-17T15:04:56.357ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6502024-10-17T15:04:56.357ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6512024-10-17T15:04:56.587ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6522024-10-17T15:04:56.587ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57131
6532024-10-17T15:04:56.594ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 291.495452ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6542024-10-17T15:04:56.595ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7105636974374909681 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6552024-10-17T15:04:56.595ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10045244912119534615 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6562024-10-17T15:04:56.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9521 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57131 req_id = 4c4d7f54-5afb-4c67-8fc5-c96f7b1f7b18 response_code = 200 uri = /timesync
6572024-10-17T15:04:56.598ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.280878127 } file = sled-agent/src/rack_setup/service.rs:660
6582024-10-17T15:04:56.598ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6592024-10-17T15:04:56.948ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6602024-10-17T15:04:56.948ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39182
6612024-10-17T15:04:56.956ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8355 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39182 req_id = a89b1aaa-ef26-4229-939e-ca7e7ebc8e2e response_code = 200 uri = /timesync
6622024-10-17T15:04:56.959ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.279905975 } file = sled-agent/src/rack_setup/service.rs:660
6632024-10-17T15:04:56.959ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6642024-10-17T15:04:57.185ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6652024-10-17T15:04:57.185ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64404
6662024-10-17T15:04:57.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8080 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64404 req_id = 264625b0-8284-4d79-8e94-f06afb8c4dd7 response_code = 200 uri = /timesync
6672024-10-17T15:04:57.197ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.279266268 } file = sled-agent/src/rack_setup/service.rs:660
6682024-10-17T15:04:57.197ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6692024-10-17T15:04:57.561ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6702024-10-17T15:04:57.561ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:61048
6712024-10-17T15:04:57.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8090 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:61048 req_id = 306fb1b6-8b7d-439c-99e5-9e3926d6e35f response_code = 200 uri = /timesync
6722024-10-17T15:04:57.574ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.275768936 } file = sled-agent/src/rack_setup/service.rs:660
6732024-10-17T15:04:57.574ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6742024-10-17T15:04:57.657ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8956846074571945561 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6752024-10-17T15:04:57.657ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6731155927220936436 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6762024-10-17T15:04:57.970ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6772024-10-17T15:04:57.971ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38172
6782024-10-17T15:04:57.975ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6792024-10-17T15:04:57.975ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6802024-10-17T15:04:57.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8565 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38172 req_id = f91d05f5-f69d-47d7-8e85-7199d461180a response_code = 200 uri = /timesync
6812024-10-17T15:04:57.982ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.274662644 } file = sled-agent/src/rack_setup/service.rs:660
6822024-10-17T15:04:57.982ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6832024-10-17T15:04:58.623ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6842024-10-17T15:04:58.623ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59668
6852024-10-17T15:04:58.631ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8495 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59668 req_id = 584e943c-b90a-44fc-a43c-bcb5eea4cb8e response_code = 200 uri = /timesync
6862024-10-17T15:04:58.634ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177498.595138, correction: 0.272655517 } file = sled-agent/src/rack_setup/service.rs:660
6872024-10-17T15:04:58.634ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6882024-10-17T15:04:59.570ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6892024-10-17T15:04:59.570ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40817
6902024-10-17T15:04:59.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8789 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40817 req_id = d9ba3a63-327d-486e-b59e-3250b88672e2 response_code = 200 uri = /timesync
6912024-10-17T15:04:59.581ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177499.5392387, correction: 0.269931197 } file = sled-agent/src/rack_setup/service.rs:660
6922024-10-17T15:04:59.581ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6932024-10-17T15:04:59.662ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1315565006152006779 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6942024-10-17T15:05:00.176ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11340141748898181460 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
6952024-10-17T15:05:00.366ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6962024-10-17T15:05:00.367ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33778
6972024-10-17T15:05:00.375ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8356 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33778 req_id = 1a031fba-984c-4d6f-a897-0b700aacacbe response_code = 200 uri = /timesync
6982024-10-17T15:05:00.377ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.267779827 } file = sled-agent/src/rack_setup/service.rs:660
6992024-10-17T15:05:00.377ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7002024-10-17T15:05:00.441ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6540327970462295184 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7012024-10-17T15:05:01.267ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7022024-10-17T15:05:01.267ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43892
7032024-10-17T15:05:01.275ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8048 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43892 req_id = c77c6748-08b4-4b37-84a3-757bf5ef6b79 response_code = 200 uri = /timesync
7042024-10-17T15:05:01.278ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, ref_time: 1729177495.0592434, correction: 0.264740258 } file = sled-agent/src/rack_setup/service.rs:660
7052024-10-17T15:05:01.278ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7062024-10-17T15:05:02.753ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7072024-10-17T15:05:02.753ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54281
7082024-10-17T15:05:02.762ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8188 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54281 req_id = f55eab0c-fdf5-433b-b466-2d59529bc050 response_code = 200 uri = /timesync
7092024-10-17T15:05:02.764ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177502.2284918, correction: 0.260393709 } file = sled-agent/src/rack_setup/service.rs:660
7102024-10-17T15:05:02.764ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7112024-10-17T15:05:04.115ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7122024-10-17T15:05:04.115ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42839
7132024-10-17T15:05:04.123ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8521 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42839 req_id = 5ac1dbea-691b-439c-9842-e8fd55741b02 response_code = 200 uri = /timesync
7142024-10-17T15:05:04.126ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177503.3844535, correction: 0.25673613 } file = sled-agent/src/rack_setup/service.rs:660
7152024-10-17T15:05:04.126ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7162024-10-17T15:05:05.106ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7172024-10-17T15:05:05.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50890
7182024-10-17T15:05:05.115ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8206 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50890 req_id = ccfe1158-e85a-4655-bf13-24b00fde5027 response_code = 200 uri = /timesync
7192024-10-17T15:05:05.117ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177503.8463707, correction: 0.254144758 } file = sled-agent/src/rack_setup/service.rs:660
7202024-10-17T15:05:05.117ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7212024-10-17T15:05:06.623ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7105636974374909681 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7222024-10-17T15:05:06.623ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10045244912119534615 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7232024-10-17T15:05:06.766ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56069
7242024-10-17T15:05:06.767ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:56069 req_id = 754a4cc2-677c-4ec3-bbc3-21eaa96fceff response_code = 200 uri = /baseboard
7252024-10-17T15:05:06.825ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7262024-10-17T15:05:06.826ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59471
7272024-10-17T15:05:06.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8370 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59471 req_id = 1f65bd4d-8ff8-4ddc-bd65-8a412f4bd9b3 response_code = 200 uri = /timesync
7282024-10-17T15:05:06.837ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177506.1673646, correction: 0.249380678 } file = sled-agent/src/rack_setup/service.rs:660
7292024-10-17T15:05:06.837ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7302024-10-17T15:05:07.686ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8956846074571945561 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7312024-10-17T15:05:07.687ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6731155927220936436 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7322024-10-17T15:05:08.004ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7332024-10-17T15:05:08.004ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7342024-10-17T15:05:09.691ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1315565006152006779 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7352024-10-17T15:05:09.907ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7362024-10-17T15:05:09.908ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:51041
7372024-10-17T15:05:09.915ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7632 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:51041 req_id = cc734e8d-e60c-4927-a685-f684a10f5078 response_code = 200 uri = /timesync
7382024-10-17T15:05:09.918ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177509.6340573, correction: 0.241052225 } file = sled-agent/src/rack_setup/service.rs:660
7392024-10-17T15:05:09.918ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7402024-10-17T15:05:10.205ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11340141748898181460 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7412024-10-17T15:05:10.469ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6540327970462295184 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7422024-10-17T15:05:11.657ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7432024-10-17T15:05:11.657ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50973
7442024-10-17T15:05:11.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8408 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50973 req_id = c6e44696-e975-4eeb-8e2e-0b83d5838224 response_code = 200 uri = /timesync
7452024-10-17T15:05:11.667ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177511.0131202, correction: 0.236287147 } file = sled-agent/src/rack_setup/service.rs:660
7462024-10-17T15:05:11.667ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7472024-10-17T15:05:11.962ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 560.875498ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7482024-10-17T15:05:14.492ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7492024-10-17T15:05:14.492ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60036
7502024-10-17T15:05:14.501ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8668 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60036 req_id = 38b5df30-2003-43b8-bee7-a65873c350c9 response_code = 200 uri = /timesync
7512024-10-17T15:05:14.503ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177513.321599, correction: 0.228742346 } file = sled-agent/src/rack_setup/service.rs:660
7522024-10-17T15:05:14.503ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7532024-10-17T15:05:16.652ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 7105636974374909681 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7542024-10-17T15:05:16.652ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 10045244912119534615 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7552024-10-17T15:05:17.715ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8956846074571945561 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7562024-10-17T15:05:17.715ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6731155927220936436 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7572024-10-17T15:05:17.758ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7582024-10-17T15:05:17.759ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45950
7592024-10-17T15:05:17.767ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8698 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45950 req_id = 24bb4b33-ba51-4805-ac74-3bb91ff39f7e response_code = 200 uri = /timesync
7602024-10-17T15:05:17.770ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177517.2785645, correction: 0.219883829 } file = sled-agent/src/rack_setup/service.rs:660
7612024-10-17T15:05:17.770ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7622024-10-17T15:05:18.033ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7632024-10-17T15:05:18.033ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7642024-10-17T15:05:19.720ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1315565006152006779 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7652024-10-17T15:05:20.233ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11340141748898181460 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7662024-10-17T15:05:20.497ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6540327970462295184 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7672024-10-17T15:05:22.165ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7682024-10-17T15:05:22.166ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62177
7692024-10-17T15:05:22.174ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8702 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:62177 req_id = 5e57b726-4df1-4f6f-811a-f378ed1e250e response_code = 200 uri = /timesync
7702024-10-17T15:05:22.177ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177521.4873166, correction: 0.208049923 } file = sled-agent/src/rack_setup/service.rs:660
7712024-10-17T15:05:22.177ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7722024-10-17T15:05:26.680ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 7105636974374909681 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7732024-10-17T15:05:26.681ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 10045244912119534615 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7742024-10-17T15:05:27.047ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7752024-10-17T15:05:27.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48443
7762024-10-17T15:05:27.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9544 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48443 req_id = 3c884986-437c-406c-a6cc-8546656461f5 response_code = 200 uri = /timesync
7772024-10-17T15:05:27.059ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177526.6874163, correction: 0.19482258 } file = sled-agent/src/rack_setup/service.rs:660
7782024-10-17T15:05:27.059ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7792024-10-17T15:05:27.602ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.003202678s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7802024-10-17T15:05:27.745ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8956846074571945561 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7812024-10-17T15:05:27.745ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6731155927220936436 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7822024-10-17T15:05:28.062ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7832024-10-17T15:05:28.062ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7842024-10-17T15:05:29.749ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 1315565006152006779 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7852024-10-17T15:05:30.262ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 11340141748898181460 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7862024-10-17T15:05:30.526ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6540327970462295184 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7872024-10-17T15:05:30.965ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7882024-10-17T15:05:30.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47113
7892024-10-17T15:05:30.974ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47113 req_id = e597a009-7380-4f02-a43b-d8f4f7c72c2e response_code = 200 uri = /timesync
7902024-10-17T15:05:30.977ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177530.1882467, correction: 0.184217855 } file = sled-agent/src/rack_setup/service.rs:660
7912024-10-17T15:05:30.977ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
7922024-10-17T15:05:36.883ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34039
7932024-10-17T15:05:36.883ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 40 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:34039 req_id = 369d1ea6-e184-4765-896b-aaccb22d4181 response_code = 200 uri = /baseboard
7942024-10-17T15:05:37.774ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8956846074571945561 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7952024-10-17T15:05:37.774ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6731155927220936436 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7962024-10-17T15:05:38.091ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7972024-10-17T15:05:38.091ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
7982024-10-17T15:05:39.463ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
7992024-10-17T15:05:39.463ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45617
8002024-10-17T15:05:39.472ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9114 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45617 req_id = 2b8c7a5c-8813-4c57-9ea3-db33bb4fc703 response_code = 200 uri = /timesync
8012024-10-17T15:05:39.474ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177536.989826, correction: 0.161245376 } file = sled-agent/src/rack_setup/service.rs:660
8022024-10-17T15:05:39.474ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8032024-10-17T15:05:39.778ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 1315565006152006779 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8042024-10-17T15:05:40.290ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 11340141748898181460 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8052024-10-17T15:05:40.555ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 6540327970462295184 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8062024-10-17T15:05:43.684ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.80997367s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8072024-10-17T15:05:47.803ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 8956846074571945561 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8082024-10-17T15:05:47.803ZERROSledAgent (kstat-sampler-worker): expiring kstat after too many failures error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:485 id = 6731155927220936436 reason = Duration(60s) sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8092024-10-17T15:05:48.119ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8102024-10-17T15:05:48.120ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8112024-10-17T15:05:53.574ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8122024-10-17T15:05:53.575ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40406
8132024-10-17T15:05:53.583ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8463 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40406 req_id = f235a7b5-0528-4c19-a595-9ad0c1a07497 response_code = 200 uri = /timesync
8142024-10-17T15:05:53.586ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177551.923087, correction: 0.123183049 } file = sled-agent/src/rack_setup/service.rs:660
8152024-10-17T15:05:53.586ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8162024-10-17T15:05:58.148ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8172024-10-17T15:05:58.149ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8182024-10-17T15:06:00.574ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.12057534s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8192024-10-17T15:06:07.001ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8202024-10-17T15:06:07.001ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41022
8212024-10-17T15:06:07.001ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54006
8222024-10-17T15:06:07.001ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:41022 req_id = ed5fed45-90e0-4572-aa13-cd0614e5945d response_code = 200 uri = /baseboard
8232024-10-17T15:06:07.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8849 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54006 req_id = 0ea2ed7a-fa94-4143-986c-6bdf3b813689 response_code = 200 uri = /timesync
8242024-10-17T15:06:07.014ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177565.3185914, correction: 0.086897105 } file = sled-agent/src/rack_setup/service.rs:660
8252024-10-17T15:06:07.014ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8262024-10-17T15:06:08.177ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8272024-10-17T15:06:08.177ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8282024-10-17T15:06:14.222ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8292024-10-17T15:06:14.222ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38330
8302024-10-17T15:06:14.231ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8681 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38330 req_id = efc8d7d0-ec3b-42fb-bf52-4666b5e5fa59 response_code = 200 uri = /timesync
8312024-10-17T15:06:14.233ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177573.9599154, correction: 0.067371793 } file = sled-agent/src/rack_setup/service.rs:660
8322024-10-17T15:06:14.233ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
8332024-10-17T15:06:18.206ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 8053284982680951072 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8342024-10-17T15:06:18.206ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15297487321759452715 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8352024-10-17T15:06:18.778ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 9.571795957s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8362024-10-17T15:06:22.794ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
8372024-10-17T15:06:22.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49635
8382024-10-17T15:06:22.816ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 21072 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49635 req_id = c5f13ac8-99c5-40da-80b8-59d85aa7cf69 response_code = 200 uri = /timesync
8392024-10-17T15:06:22.816ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:26.680798325Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 7105636974374909681 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8402024-10-17T15:06:22.818ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:40.555070207Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 6540327970462295184 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8412024-10-17T15:06:22.818ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:47.803309357Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 8956846074571945561 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8422024-10-17T15:06:22.819ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:47.803599324Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 6731155927220936436 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8432024-10-17T15:06:22.819ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:40.290975549Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 11340141748898181460 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8442024-10-17T15:06:22.819ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:26.681416534Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 10045244912119534615 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8452024-10-17T15:06:22.819ZWARNSledAgent (kstat-sampler-worker): replacing expired kstat target error = NoSuchKstat expiration_reason = Duration(60s) expired_at = 2024-10-17T15:05:39.778630129Z file = oximeter/instruments/src/kstat/sampler.rs:1028 id = 1315565006152006779 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
8462024-10-17T15:06:22.820ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1729177581.3700037, correction: 0.044225611 } file = sled-agent/src/rack_setup/service.rs:660
8472024-10-17T15:06:22.820ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1298
8482024-10-17T15:06:22.849ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 764d7304-941c-47d5-bd71-20d679249cf8 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 703a01d4-fb0e-4834-936c-a729fd81d47b (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 24c32b69-efe8-479e-ba18-19f8c4fd316f (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: c7a31b26-8275-490c-9a47-724548eb0d53, kind: Service { id: 24c32b69-efe8-479e-ba18-19f8c4fd316f }, name: Name("ntp-24c32b69-efe8-479e-ba18-19f8c4fd316f"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 52])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 32cb6f24-454a-4d30-8647-45df4cd85bbb (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } } } }, OmicronZoneConfig { id: 491f0d0d-6d63-42e0-9c3f-b51f48c45ec5 (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } } } }, OmicronZoneConfig { id: 43749833-fb79-42cb-aa86-337230b31b95 (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6f926079-175a-448b-a901-3ffe573980d6 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 92eabd86-71b3-4a80-bff4-cd762a556252 (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8492024-10-17T15:06:22.850ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62600
8502024-10-17T15:06:22.850ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8512024-10-17T15:06:22.850ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }
8522024-10-17T15:06:22.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8532024-10-17T15:06:22.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }
8542024-10-17T15:06:22.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8552024-10-17T15:06:22.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8562024-10-17T15:06:22.908ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32cb6f24-454a-4d30-8647-45df4cd85bbb, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8572024-10-17T15:06:22.908ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 32cb6f24-454a-4d30-8647-45df4cd85bbb, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Cockroach }, responder: ... }
8582024-10-17T15:06:22.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 491f0d0d-6d63-42e0-9c3f-b51f48c45ec5, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8592024-10-17T15:06:22.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 491f0d0d-6d63-42e0-9c3f-b51f48c45ec5, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: Cockroach }, responder: ... }
8602024-10-17T15:06:23.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 43749833-fb79-42cb-aa86-337230b31b95, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8612024-10-17T15:06:23.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 43749833-fb79-42cb-aa86-337230b31b95, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8622024-10-17T15:06:23.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f926079-175a-448b-a901-3ffe573980d6, dataset_name: DatasetName { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8632024-10-17T15:06:23.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f926079-175a-448b-a901-3ffe573980d6, dataset_name: DatasetName { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8642024-10-17T15:06:23.149ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 92eabd86-71b3-4a80-bff4-cd762a556252, dataset_name: DatasetName { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8652024-10-17T15:06:23.149ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 92eabd86-71b3-4a80-bff4-cd762a556252, dataset_name: DatasetName { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8662024-10-17T15:06:23.208ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8672024-10-17T15:06:23.208ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:177
8682024-10-17T15:06:23.209ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:177
8692024-10-17T15:06:23.209ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8702024-10-17T15:06:23.289ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8712024-10-17T15:06:23.289ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8722024-10-17T15:06:23.289ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8732024-10-17T15:06:23.289ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8742024-10-17T15:06:23.289ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8752024-10-17T15:06:23.382ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb file = illumos-utils/src/zone.rs:316
8762024-10-17T15:06:23.386ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5 file = illumos-utils/src/zone.rs:316
8772024-10-17T15:06:23.391ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95 file = illumos-utils/src/zone.rs:316
8782024-10-17T15:06:23.395ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6 file = illumos-utils/src/zone.rs:316
8792024-10-17T15:06:23.400ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252 file = illumos-utils/src/zone.rs:316
8802024-10-17T15:06:23.404ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb file = illumos-utils/src/zone.rs:354
8812024-10-17T15:06:23.409ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5 file = illumos-utils/src/zone.rs:354
8822024-10-17T15:06:23.414ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95 file = illumos-utils/src/zone.rs:354
8832024-10-17T15:06:23.419ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6 file = illumos-utils/src/zone.rs:354
8842024-10-17T15:06:23.423ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252 file = illumos-utils/src/zone.rs:354
8852024-10-17T15:06:26.290ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='491f0d0d-6d63-42e0-9c3f-b51f48c45ec5'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8862024-10-17T15:06:26.290ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='43749833-fb79-42cb-aa86-337230b31b95'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8872024-10-17T15:06:26.290ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='6f926079-175a-448b-a901-3ffe573980d6'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8882024-10-17T15:06:26.290ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
8892024-10-17T15:06:26.296ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
8902024-10-17T15:06:26.300ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
8912024-10-17T15:06:26.426ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='92eabd86-71b3-4a80-bff4-cd762a556252'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8922024-10-17T15:06:26.427ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='32cb6f24-454a-4d30-8647-45df4cd85bbb'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
8932024-10-17T15:06:26.427ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
8942024-10-17T15:06:26.432ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
8952024-10-17T15:06:26.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 47.721369ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
8962024-10-17T15:06:26.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 39.274063ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
8972024-10-17T15:06:26.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 68.386259ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
8982024-10-17T15:06:26.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 62.17883ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
8992024-10-17T15:06:26.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 105.329867ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9002024-10-17T15:06:26.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 170.666554ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9012024-10-17T15:06:27.115ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 140.69236ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9022024-10-17T15:06:27.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 297.331969ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9032024-10-17T15:06:27.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 340.383165ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9042024-10-17T15:06:27.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 219.448882ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9052024-10-17T15:06:27.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 66.750802ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9062024-10-17T15:06:27.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 34.362141ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9072024-10-17T15:06:27.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 136.856565ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9082024-10-17T15:06:27.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 75.422617ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9092024-10-17T15:06:27.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 285.249042ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9102024-10-17T15:06:27.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.167093841s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9112024-10-17T15:06:27.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 556.154555ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9122024-10-17T15:06:27.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 200.471982ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9132024-10-17T15:06:27.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 143.619369ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9142024-10-17T15:06:27.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 1.086565631s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9152024-10-17T15:06:27.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 410.896885ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9162024-10-17T15:06:28.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 452.563018ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9172024-10-17T15:06:28.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 985.586258ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9182024-10-17T15:06:28.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 419.954031ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9192024-10-17T15:06:28.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 744.84484ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9202024-10-17T15:06:28.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.126249581s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9212024-10-17T15:06:28.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 1.496710149s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9222024-10-17T15:06:29.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 606.452511ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9232024-10-17T15:06:29.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 1.04727466s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9242024-10-17T15:06:29.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 1.490073496s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9252024-10-17T15:06:29.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 1.143763274s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9262024-10-17T15:06:29.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 894.864795ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9272024-10-17T15:06:30.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 1.019497659s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9282024-10-17T15:06:30.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 1.454151014s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9292024-10-17T15:06:30.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 1.400675489s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9302024-10-17T15:06:30.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.260071559s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9312024-10-17T15:06:30.874ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 1.485009791s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9322024-10-17T15:06:31.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 1.112090922s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9332024-10-17T15:06:31.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 1.012059412s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9342024-10-17T15:06:32.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 767.471594ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9352024-10-17T15:06:32.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 1.07197125s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9362024-10-17T15:06:32.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 1.242178158s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9372024-10-17T15:06:32.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 763.509925ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9382024-10-17T15:06:32.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.307842073s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9392024-10-17T15:06:32.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 787.931597ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9402024-10-17T15:06:33.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 891.12466ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9412024-10-17T15:06:33.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 578.547402ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9422024-10-17T15:06:33.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5") failed: Property not found. retry in 1.441207474s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9432024-10-17T15:06:33.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 943.609987ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9442024-10-17T15:06:33.906ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6") failed: Property not found. retry in 1.166227226s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9452024-10-17T15:06:34.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 789.635882ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9462024-10-17T15:06:34.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 764.799825ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9472024-10-17T15:06:34.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252") failed: Property not found. retry in 1.120077046s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9482024-10-17T15:06:34.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 711.839352ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9492024-10-17T15:06:35.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 558.700189ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9502024-10-17T15:06:35.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 787.958046ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9512024-10-17T15:06:35.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 1.148092982s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9522024-10-17T15:06:36.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.24567579s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9532024-10-17T15:06:37.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 991.202171ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9542024-10-17T15:06:37.128ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38799
9552024-10-17T15:06:37.128ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:38799 req_id = 31d5356f-1b4d-4110-b55e-b7b3b5e9f9ff response_code = 200 uri = /baseboard
9562024-10-17T15:06:37.653ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 856.715379ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9572024-10-17T15:06:38.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 831.895351ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9582024-10-17T15:06:38.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.297834861s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9592024-10-17T15:06:38.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 925.736013ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9602024-10-17T15:06:39.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 847.998875ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9612024-10-17T15:06:39.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 808.474275ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9622024-10-17T15:06:40.652ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 745.683465ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9632024-10-17T15:06:40.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 920.772934ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9642024-10-17T15:06:41.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.100320345s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9652024-10-17T15:06:41.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 1.406269225s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9662024-10-17T15:06:42.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.266999828s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9672024-10-17T15:06:43.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 805.634015ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9682024-10-17T15:06:43.510ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 10.641302752s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
9692024-10-17T15:06:43.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95") failed: Property not found. retry in 1.194014808s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9702024-10-17T15:06:43.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 790.21133ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9712024-10-17T15:06:44.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb") failed: Property not found. retry in 641.680118ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9722024-10-17T15:06:45.417ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_32cb6f24-454a-4d30-8647-45df4cd85bbb
9732024-10-17T15:06:45.417ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_491f0d0d-6d63-42e0-9c3f-b51f48c45ec5
9742024-10-17T15:06:45.417ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_43749833-fb79-42cb-aa86-337230b31b95
9752024-10-17T15:06:45.417ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_6f926079-175a-448b-a901-3ffe573980d6
9762024-10-17T15:06:45.417ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_92eabd86-71b3-4a80-bff4-cd762a556252
9772024-10-17T15:06:45.417ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9782024-10-17T15:06:45.418ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
9792024-10-17T15:06:45.461ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22567629 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:62600 req_id = 13b29427-61a3-486e-8efb-b43411512cf3 response_code = 204 uri = /omicron-zones
9802024-10-17T15:06:45.494ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38684
9812024-10-17T15:06:45.525ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3720
9822024-10-17T15:06:51.584ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3738
9832024-10-17T15:07:07.165ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33578
9842024-10-17T15:07:07.165ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 30 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:33578 req_id = bb434eab-d329-4a9f-b3b1-d15b60b8d914 response_code = 200 uri = /baseboard
9852024-10-17T15:07:08.085ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3761
9862024-10-17T15:07:08.085ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22591457 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:38684 req_id = 3fde8f18-0e25-48b5-8464-71f5a4623175 response_code = 204 uri = /cockroachdb
9872024-10-17T15:07:08.116ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 764d7304-941c-47d5-bd71-20d679249cf8 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 703a01d4-fb0e-4834-936c-a729fd81d47b (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 24c32b69-efe8-479e-ba18-19f8c4fd316f (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: c7a31b26-8275-490c-9a47-724548eb0d53, kind: Service { id: 24c32b69-efe8-479e-ba18-19f8c4fd316f }, name: Name("ntp-24c32b69-efe8-479e-ba18-19f8c4fd316f"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 52])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.1.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 32cb6f24-454a-4d30-8647-45df4cd85bbb (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } } } }, OmicronZoneConfig { id: 491f0d0d-6d63-42e0-9c3f-b51f48c45ec5 (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } } } }, OmicronZoneConfig { id: 43749833-fb79-42cb-aa86-337230b31b95 (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6f926079-175a-448b-a901-3ffe573980d6 (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 92eabd86-71b3-4a80-bff4-cd762a556252 (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External } } } }, OmicronZoneConfig { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.1.164:53, nic: NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.1.165:53, nic: NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 9125336f-8bba-4da8-a415-ccae7d60d01b (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.1.166, nic: NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 282d1bc9-0cd4-4794-93fb-67f38865892f (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.1.167, nic: NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.1.168, nic: NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 15b75261-ed7f-4731-8d7c-026faffa306d (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: f719d96b-fc30-410a-9263-e31911b41872 (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } } } }, OmicronZoneConfig { id: 310d3292-985c-40d5-bdf7-ab20ee75d725 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: d91e68f0-386e-4018-acb6-b454eb8d2c78 (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 3f0e9f53-0a32-4566-9bce-284027b87815 (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: b794f397-7d13-48f2-bdc7-19ec2afdbebb (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External } } } }, OmicronZoneConfig { id: b5b2a17f-0793-48a1-96fe-fd8f74634fc1 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External } } } }, OmicronZoneConfig { id: a25f7721-b451-46d8-90eb-75e61b459dc4 (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External } } } }, OmicronZoneConfig { id: a4097c34-ce96-4b8a-8481-62ca4e7a5dd5 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External } } } }, OmicronZoneConfig { id: 74b5cdc0-3a16-484b-9fb3-b7f86509f614 (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External } } } }, OmicronZoneConfig { id: 12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External } } } }, OmicronZoneConfig { id: 283be9d0-c6b9-4a8a-b8aa-c7975b11496e (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool), kind: External } } } }, OmicronZoneConfig { id: b8c2d4d5-8168-42c0-b626-1f4fdd302ca8 (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool), kind: External } } } }, OmicronZoneConfig { id: b24a0bb5-2498-4017-b9e7-384727507bb4 (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9882024-10-17T15:07:08.116ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35325
9892024-10-17T15:07:08.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9902024-10-17T15:07:08.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 96c809da-b476-4ddd-bbdc-51baf4d3e56f, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: InternalDns }, responder: ... }
9912024-10-17T15:07:08.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9922024-10-17T15:07:08.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 764d7304-941c-47d5-bd71-20d679249cf8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: InternalDns }, responder: ... }
9932024-10-17T15:07:08.157ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9942024-10-17T15:07:08.157ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 703a01d4-fb0e-4834-936c-a729fd81d47b, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9952024-10-17T15:07:08.175ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 32cb6f24-454a-4d30-8647-45df4cd85bbb, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9962024-10-17T15:07:08.175ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 32cb6f24-454a-4d30-8647-45df4cd85bbb, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Cockroach }, responder: ... }
9972024-10-17T15:07:08.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 491f0d0d-6d63-42e0-9c3f-b51f48c45ec5, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9982024-10-17T15:07:08.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 491f0d0d-6d63-42e0-9c3f-b51f48c45ec5, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: Cockroach }, responder: ... }
9992024-10-17T15:07:08.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 43749833-fb79-42cb-aa86-337230b31b95, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10002024-10-17T15:07:08.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 43749833-fb79-42cb-aa86-337230b31b95, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10012024-10-17T15:07:08.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f926079-175a-448b-a901-3ffe573980d6, dataset_name: DatasetName { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10022024-10-17T15:07:08.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f926079-175a-448b-a901-3ffe573980d6, dataset_name: DatasetName { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10032024-10-17T15:07:08.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 92eabd86-71b3-4a80-bff4-cd762a556252, dataset_name: DatasetName { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10042024-10-17T15:07:08.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 92eabd86-71b3-4a80-bff4-cd762a556252, dataset_name: DatasetName { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10052024-10-17T15:07:08.267ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d8aecb7f-3c94-4690-8de6-10380087d1f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10062024-10-17T15:07:08.267ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: d8aecb7f-3c94-4690-8de6-10380087d1f0, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10072024-10-17T15:07:08.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 869dbbb9-e780-4452-80fd-27d18cfc03d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10082024-10-17T15:07:08.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 869dbbb9-e780-4452-80fd-27d18cfc03d9, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10092024-10-17T15:07:08.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f719d96b-fc30-410a-9263-e31911b41872, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
10102024-10-17T15:07:08.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f719d96b-fc30-410a-9263-e31911b41872, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10112024-10-17T15:07:08.534ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b794f397-7d13-48f2-bdc7-19ec2afdbebb, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10122024-10-17T15:07:08.534ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b794f397-7d13-48f2-bdc7-19ec2afdbebb, dataset_name: DatasetName { pool_name: ZpoolName { id: 86d90d1a-236b-48f7-9d13-e0e08c6f416e (zpool), kind: External }, kind: Crucible }, responder: ... }
10132024-10-17T15:07:08.633ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b5b2a17f-0793-48a1-96fe-fd8f74634fc1, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10142024-10-17T15:07:08.633ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b5b2a17f-0793-48a1-96fe-fd8f74634fc1, dataset_name: DatasetName { pool_name: ZpoolName { id: 6bbd0bbe-4092-4d91-b621-d8484bd8935a (zpool), kind: External }, kind: Crucible }, responder: ... }
10152024-10-17T15:07:08.728ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a25f7721-b451-46d8-90eb-75e61b459dc4, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10162024-10-17T15:07:08.728ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a25f7721-b451-46d8-90eb-75e61b459dc4, dataset_name: DatasetName { pool_name: ZpoolName { id: 13507129-93b4-4cdb-aa3e-5a74ab4d7515 (zpool), kind: External }, kind: Crucible }, responder: ... }
10172024-10-17T15:07:08.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a4097c34-ce96-4b8a-8481-62ca4e7a5dd5, dataset_name: DatasetName { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10182024-10-17T15:07:08.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a4097c34-ce96-4b8a-8481-62ca4e7a5dd5, dataset_name: DatasetName { pool_name: ZpoolName { id: d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0 (zpool), kind: External }, kind: Crucible }, responder: ... }
10192024-10-17T15:07:08.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 74b5cdc0-3a16-484b-9fb3-b7f86509f614, dataset_name: DatasetName { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10202024-10-17T15:07:08.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 74b5cdc0-3a16-484b-9fb3-b7f86509f614, dataset_name: DatasetName { pool_name: ZpoolName { id: df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f (zpool), kind: External }, kind: Crucible }, responder: ... }
10212024-10-17T15:07:09.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a, dataset_name: DatasetName { pool_name: ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10222024-10-17T15:07:09.059ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a, dataset_name: DatasetName { pool_name: ZpoolName { id: e5d462a9-5343-4c50-9dc4-2adbb2b66d5e (zpool), kind: External }, kind: Crucible }, responder: ... }
10232024-10-17T15:07:09.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 283be9d0-c6b9-4a8a-b8aa-c7975b11496e, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10242024-10-17T15:07:09.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 283be9d0-c6b9-4a8a-b8aa-c7975b11496e, dataset_name: DatasetName { pool_name: ZpoolName { id: 0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf (zpool), kind: External }, kind: Crucible }, responder: ... }
10252024-10-17T15:07:09.188ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 38.580240671s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
10262024-10-17T15:07:09.210ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b8c2d4d5-8168-42c0-b626-1f4fdd302ca8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10272024-10-17T15:07:09.210ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b8c2d4d5-8168-42c0-b626-1f4fdd302ca8, dataset_name: DatasetName { pool_name: ZpoolName { id: 6dc43632-8304-473f-ba75-413bbd171b4c (zpool), kind: External }, kind: Crucible }, responder: ... }
10282024-10-17T15:07:09.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b24a0bb5-2498-4017-b9e7-384727507bb4, dataset_name: DatasetName { pool_name: ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10292024-10-17T15:07:09.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b24a0bb5-2498-4017-b9e7-384727507bb4, dataset_name: DatasetName { pool_name: ZpoolName { id: 6931ef6e-c78c-4d61-ab93-92f35309d55f (zpool), kind: External }, kind: Crucible }, responder: ... }
10302024-10-17T15:07:09.365ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10312024-10-17T15:07:09.365ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:177
10322024-10-17T15:07:09.366ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:177
10332024-10-17T15:07:09.366ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10342024-10-17T15:07:09.460ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10352024-10-17T15:07:09.460ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10362024-10-17T15:07:09.485ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10372024-10-17T15:07:09.485ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10382024-10-17T15:07:09.532ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10392024-10-17T15:07:09.532ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10402024-10-17T15:07:09.534ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10412024-10-17T15:07:09.535ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10422024-10-17T15:07:09.535ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10432024-10-17T15:07:09.535ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10442024-10-17T15:07:09.733ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10452024-10-17T15:07:09.733ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10462024-10-17T15:07:09.733ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10472024-10-17T15:07:09.733ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10482024-10-17T15:07:09.733ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10492024-10-17T15:07:09.745ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10502024-10-17T15:07:09.745ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10512024-10-17T15:07:09.747ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10522024-10-17T15:07:09.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10532024-10-17T15:07:09.758ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d file = illumos-utils/src/zone.rs:316
10542024-10-17T15:07:09.763ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4 file = illumos-utils/src/zone.rs:316
10552024-10-17T15:07:09.768ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815 file = illumos-utils/src/zone.rs:316
10562024-10-17T15:07:09.772ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5 file = illumos-utils/src/zone.rs:316
10572024-10-17T15:07:09.777ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614 file = illumos-utils/src/zone.rs:316
10582024-10-17T15:07:09.782ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872 file = illumos-utils/src/zone.rs:316
10592024-10-17T15:07:09.787ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725 file = illumos-utils/src/zone.rs:316
10602024-10-17T15:07:09.791ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb file = illumos-utils/src/zone.rs:316
10612024-10-17T15:07:09.914ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d file = illumos-utils/src/zone.rs:354
10622024-10-17T15:07:09.918ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4 file = illumos-utils/src/zone.rs:354
10632024-10-17T15:07:09.923ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815 file = illumos-utils/src/zone.rs:354
10642024-10-17T15:07:09.927ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5 file = illumos-utils/src/zone.rs:354
10652024-10-17T15:07:09.932ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614 file = illumos-utils/src/zone.rs:354
10662024-10-17T15:07:09.937ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872 file = illumos-utils/src/zone.rs:354
10672024-10-17T15:07:09.941ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725 file = illumos-utils/src/zone.rs:354
10682024-10-17T15:07:09.946ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb file = illumos-utils/src/zone.rs:354
10692024-10-17T15:07:09.950ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a file = illumos-utils/src/zone.rs:316
10702024-10-17T15:07:09.956ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e file = illumos-utils/src/zone.rs:316
10712024-10-17T15:07:09.961ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8 file = illumos-utils/src/zone.rs:316
10722024-10-17T15:07:09.966ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4 file = illumos-utils/src/zone.rs:316
10732024-10-17T15:07:09.971ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78 file = illumos-utils/src/zone.rs:316
10742024-10-17T15:07:09.977ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1 file = illumos-utils/src/zone.rs:316
10752024-10-17T15:07:09.983ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a file = illumos-utils/src/zone.rs:354
10762024-10-17T15:07:09.990ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e file = illumos-utils/src/zone.rs:354
10772024-10-17T15:07:09.995ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8 file = illumos-utils/src/zone.rs:354
10782024-10-17T15:07:10.001ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4 file = illumos-utils/src/zone.rs:354
10792024-10-17T15:07:10.008ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78 file = illumos-utils/src/zone.rs:354
10802024-10-17T15:07:10.014ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1 file = illumos-utils/src/zone.rs:354
10812024-10-17T15:07:10.079ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10822024-10-17T15:07:10.117ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10832024-10-17T15:07:10.157ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10842024-10-17T15:07:10.272ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10852024-10-17T15:07:10.325ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
10862024-10-17T15:07:10.327ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10872024-10-17T15:07:10.327ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10882024-10-17T15:07:10.377ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 177, 199]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10892024-10-17T15:07:10.377ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
10902024-10-17T15:07:10.379ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10912024-10-17T15:07:10.379ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10922024-10-17T15:07:10.425ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 178, 199]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
10932024-10-17T15:07:10.425ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
10942024-10-17T15:07:10.428ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
10952024-10-17T15:07:10.428ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
10962024-10-17T15:07:10.474ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 230, 242]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
10972024-10-17T15:07:10.474ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
10982024-10-17T15:07:10.475ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:83
10992024-10-17T15:07:10.475ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:90
11002024-10-17T15:07:10.475ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:83
11012024-10-17T15:07:10.475ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:90
11022024-10-17T15:07:10.476ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11032024-10-17T15:07:10.476ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11042024-10-17T15:07:10.530ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 160, 83]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11052024-10-17T15:07:10.530ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
11062024-10-17T15:07:10.532ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11072024-10-17T15:07:10.532ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11082024-10-17T15:07:10.573ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 197, 237]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11092024-10-17T15:07:10.573ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
11102024-10-17T15:07:10.573ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.166 file = clients/dpd-client/src/lib.rs:118
11112024-10-17T15:07:10.575ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.164 file = clients/dpd-client/src/lib.rs:118
11122024-10-17T15:07:10.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11132024-10-17T15:07:10.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11142024-10-17T15:07:10.642ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:83
11152024-10-17T15:07:10.642ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:90
11162024-10-17T15:07:10.642ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b file = illumos-utils/src/zone.rs:316
11172024-10-17T15:07:10.650ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:83
11182024-10-17T15:07:10.650ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:90
11192024-10-17T15:07:10.650ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:83
11202024-10-17T15:07:10.650ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:90
11212024-10-17T15:07:10.650ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.167 file = clients/dpd-client/src/lib.rs:118
11222024-10-17T15:07:10.653ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11232024-10-17T15:07:10.653ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.168 file = clients/dpd-client/src/lib.rs:118
11242024-10-17T15:07:10.655ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.1.165 file = clients/dpd-client/src/lib.rs:118
11252024-10-17T15:07:10.657ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11262024-10-17T15:07:10.657ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11272024-10-17T15:07:10.691ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b file = illumos-utils/src/zone.rs:354
11282024-10-17T15:07:10.700ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0 file = illumos-utils/src/zone.rs:316
11292024-10-17T15:07:10.849ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f file = illumos-utils/src/zone.rs:316
11302024-10-17T15:07:10.889ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0 file = illumos-utils/src/zone.rs:354
11312024-10-17T15:07:10.915ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64 file = illumos-utils/src/zone.rs:316
11322024-10-17T15:07:10.950ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9 file = illumos-utils/src/zone.rs:316
11332024-10-17T15:07:10.970ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f file = illumos-utils/src/zone.rs:354
11342024-10-17T15:07:11.010ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64 file = illumos-utils/src/zone.rs:354
11352024-10-17T15:07:11.025ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9 file = illumos-utils/src/zone.rs:354
11362024-10-17T15:07:12.509ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_13507129-93b4-4cdb-aa3e-5a74ab4d7515/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='8dfec94c-7545-4dce-aa26-84ad7ef9a4ba'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11372024-10-17T15:07:12.516ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11382024-10-17T15:07:12.664ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11392024-10-17T15:07:12.664ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
11402024-10-17T15:07:13.456ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ce88d789-3ca8-4075-aab3-cd9273ac90f3'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11412024-10-17T15:07:13.456ZINFOSledAgent (ServiceManager): Booting oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11422024-10-17T15:07:13.933ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='15b75261-ed7f-4731-8d7c-026faffa306d'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11432024-10-17T15:07:13.933ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
11442024-10-17T15:07:14.003ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='1439fa89-04c9-4334-8f4e-acf911563217'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11452024-10-17T15:07:14.003ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
11462024-10-17T15:07:14.398ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='998984d3-89d7-4259-ac8b-0dfe2e6fd4b6'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11472024-10-17T15:07:14.398ZINFOSledAgent (ServiceManager): Booting oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
11482024-10-17T15:07:17.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 25.199334ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11492024-10-17T15:07:17.682ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 69.762389ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
11502024-10-17T15:07:17.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 39.595041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11512024-10-17T15:07:17.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 103.402517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11522024-10-17T15:07:17.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 95.099505ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
11532024-10-17T15:07:17.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 139.919703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11542024-10-17T15:07:17.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 298.526126ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11552024-10-17T15:07:18.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 155.388115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
11562024-10-17T15:07:18.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 30.079099ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
11572024-10-17T15:07:18.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 74.122264ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
11582024-10-17T15:07:18.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 57.337027ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
11592024-10-17T15:07:18.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 177.161581ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11602024-10-17T15:07:18.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 102.216162ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
11612024-10-17T15:07:18.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 300.704109ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11622024-10-17T15:07:18.310ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 530.154426ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
11632024-10-17T15:07:18.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 241.350814ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
11642024-10-17T15:07:18.334ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 27.13862ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
11652024-10-17T15:07:18.334ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_6931ef6e-c78c-4d61-ab93-92f35309d55f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='7959f27d-68e9-4dec-8816-8c4b8d65907a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11662024-10-17T15:07:18.335ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11672024-10-17T15:07:18.335ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
11682024-10-17T15:07:18.341ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
11692024-10-17T15:07:18.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 95.329143ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
11702024-10-17T15:07:18.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 284.276003ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
11712024-10-17T15:07:18.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 233.272469ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11722024-10-17T15:07:18.500ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 148.270068ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
11732024-10-17T15:07:18.521ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_6dc43632-8304-473f-ba75-413bbd171b4c/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='eb9f6f82-f952-445e-8c22-9b8d565b5731'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11742024-10-17T15:07:18.522ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
11752024-10-17T15:07:18.530ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_6bbd0bbe-4092-4d91-b621-d8484bd8935a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='7a2f80e5-3b69-4a1d-b520-f2bd24514608'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11762024-10-17T15:07:18.530ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e5d462a9-5343-4c50-9dc4-2adbb2b66d5e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5eee9927-9881-4ec4-8bbd-aad9608e7010'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11772024-10-17T15:07:18.530ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
11782024-10-17T15:07:18.537ZINFOSledAgent (ServiceManager): Booting oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
11792024-10-17T15:07:18.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 226.092061ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
11802024-10-17T15:07:18.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 722.750129ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11812024-10-17T15:07:18.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 525.152061ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
11822024-10-17T15:07:18.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 517.555043ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
11832024-10-17T15:07:18.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 457.786911ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11842024-10-17T15:07:18.764ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11852024-10-17T15:07:18.765ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
11862024-10-17T15:07:21.844ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 738.026869ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
11872024-10-17T15:07:21.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.19961496s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
11882024-10-17T15:07:21.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.189119509s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
11892024-10-17T15:07:21.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 489.982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
11902024-10-17T15:07:21.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 772.296451ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
11912024-10-17T15:07:21.973ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_86d90d1a-236b-48f7-9d13-e0e08c6f416e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='08463f0f-bfa9-43f4-857c-4f5baaff9ec4'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11922024-10-17T15:07:21.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.152132469s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
11932024-10-17T15:07:22.020ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 28.474633ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
11942024-10-17T15:07:22.020ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
11952024-10-17T15:07:22.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 81.645371ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
11962024-10-17T15:07:22.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 37.70211ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
11972024-10-17T15:07:22.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 193.61615ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
11982024-10-17T15:07:22.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 43.063475ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
11992024-10-17T15:07:22.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.060796073s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
12002024-10-17T15:07:22.714ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 32.223138ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12012024-10-17T15:07:22.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 63.463755ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12022024-10-17T15:07:22.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 116.706862ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12032024-10-17T15:07:23.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 226.296212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
12042024-10-17T15:07:23.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 146.904018ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12052024-10-17T15:07:23.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.244323944s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
12062024-10-17T15:07:23.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 862.856591ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
12072024-10-17T15:07:23.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.433316477s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
12082024-10-17T15:07:23.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 671.886214ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
12092024-10-17T15:07:23.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 43.44529ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12102024-10-17T15:07:23.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 148.418982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12112024-10-17T15:07:23.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.107491879s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
12122024-10-17T15:07:23.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 52.509503ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12132024-10-17T15:07:23.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 231.688325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12142024-10-17T15:07:23.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 626.614902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
12152024-10-17T15:07:23.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 144.197412ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12162024-10-17T15:07:23.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.333996829s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
12172024-10-17T15:07:23.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 100.987139ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12182024-10-17T15:07:23.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 48.214123ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12192024-10-17T15:07:23.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 265.716549ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12202024-10-17T15:07:23.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 59.268038ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12212024-10-17T15:07:23.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 146.23347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12222024-10-17T15:07:23.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 125.872527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12232024-10-17T15:07:23.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 439.376052ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12242024-10-17T15:07:23.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 111.38723ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12252024-10-17T15:07:23.926ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 432.644561ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12262024-10-17T15:07:23.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 233.572261ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12272024-10-17T15:07:23.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 367.39029ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12282024-10-17T15:07:23.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 218.037831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12292024-10-17T15:07:24.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 304.928278ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12302024-10-17T15:07:24.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 607.766968ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12312024-10-17T15:07:24.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 846.758189ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12322024-10-17T15:07:24.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 712.531017ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
12332024-10-17T15:07:24.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 518.9013ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12342024-10-17T15:07:24.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 1.207111897s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
12352024-10-17T15:07:24.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 510.737077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12362024-10-17T15:07:24.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 566.267299ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12372024-10-17T15:07:24.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 480.547925ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12382024-10-17T15:07:24.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.473322348s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
12392024-10-17T15:07:24.531ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12402024-10-17T15:07:24.531ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12412024-10-17T15:07:24.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.06725448s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
12422024-10-17T15:07:24.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 66.234932ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12432024-10-17T15:07:24.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.458994703s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12442024-10-17T15:07:24.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 916.931626ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
12452024-10-17T15:07:24.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.379579081s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12462024-10-17T15:07:24.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 512.857394ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12472024-10-17T15:07:24.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.420124988s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12482024-10-17T15:07:24.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 87.207569ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12492024-10-17T15:07:24.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 856.126443ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12502024-10-17T15:07:24.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 184.899478ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12512024-10-17T15:07:25.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 830.763227ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
12522024-10-17T15:07:25.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 785.175157ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
12532024-10-17T15:07:25.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 992.211072ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
12542024-10-17T15:07:25.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.187889092s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12552024-10-17T15:07:25.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 410.867437ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12562024-10-17T15:07:25.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 558.970173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12572024-10-17T15:07:25.558ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 574.104302ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
12582024-10-17T15:07:25.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 444.151768ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12592024-10-17T15:07:25.686ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12602024-10-17T15:07:25.686ZINFOSledAgent (ServiceManager): Booting oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12612024-10-17T15:07:25.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 677.332863ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
12622024-10-17T15:07:25.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.305410332s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12632024-10-17T15:07:25.851ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.435282855s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
12642024-10-17T15:07:25.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 589.996453ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
12652024-10-17T15:07:25.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 735.418483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
12662024-10-17T15:07:25.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 33.996183ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12672024-10-17T15:07:25.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.332190969s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
12682024-10-17T15:07:25.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 837.423483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12692024-10-17T15:07:25.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 67.079394ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12702024-10-17T15:07:26.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 175.517321ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12712024-10-17T15:07:26.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 590.003406ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
12722024-10-17T15:07:26.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 710.997067ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12732024-10-17T15:07:26.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 711.597886ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
12742024-10-17T15:07:26.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 312.9014ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12752024-10-17T15:07:26.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 762.52711ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12762024-10-17T15:07:26.287ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.321212709s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12772024-10-17T15:07:26.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 570.608399ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12782024-10-17T15:07:26.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.480380696s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
12792024-10-17T15:07:26.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 791.651168ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
12802024-10-17T15:07:26.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.319173928s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
12812024-10-17T15:07:26.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 1.003267328s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12822024-10-17T15:07:26.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 1.188693141s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
12832024-10-17T15:07:26.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 838.659273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
12842024-10-17T15:07:26.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 938.841715ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12852024-10-17T15:07:26.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 1.015571669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
12862024-10-17T15:07:26.866ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 936.063308ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
12872024-10-17T15:07:26.889ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.195982347s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
12882024-10-17T15:07:27.034ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.180404121s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
12892024-10-17T15:07:27.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.221952295s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
12902024-10-17T15:07:27.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 515.057441ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
12912024-10-17T15:07:27.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 910.894986ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
12922024-10-17T15:07:27.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.231218625s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
12932024-10-17T15:07:27.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.0187038s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
12942024-10-17T15:07:27.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 664.633481ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
12952024-10-17T15:07:27.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.341276584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
12962024-10-17T15:07:27.754ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.416458682s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
12972024-10-17T15:07:27.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.426901623s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
12982024-10-17T15:07:27.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 619.586532ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
12992024-10-17T15:07:27.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 909.858724ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
13002024-10-17T15:07:27.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 838.315702ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13012024-10-17T15:07:27.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 1.244500035s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
13022024-10-17T15:07:27.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 1.127688806s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
13032024-10-17T15:07:28.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.059958402s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
13042024-10-17T15:07:28.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 530.608707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
13052024-10-17T15:07:28.243ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 640.29518ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
13062024-10-17T15:07:28.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 905.565964ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
13072024-10-17T15:07:28.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 519.947384ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
13082024-10-17T15:07:28.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.011457377s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
13092024-10-17T15:07:28.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.222243711s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
13102024-10-17T15:07:28.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 574.319749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13112024-10-17T15:07:28.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 885.640909ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
13122024-10-17T15:07:28.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 800.030517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
13132024-10-17T15:07:28.835ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.17748469s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
13142024-10-17T15:07:28.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.40967907s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
13152024-10-17T15:07:28.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.497228584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
13162024-10-17T15:07:28.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 596.09582ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
13172024-10-17T15:07:29.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 814.963187ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
13182024-10-17T15:07:29.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 672.33077ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
13192024-10-17T15:07:29.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.263092518s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
13202024-10-17T15:07:29.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.30716508s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
13212024-10-17T15:07:29.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 971.055605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
13222024-10-17T15:07:29.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 999.683857ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
13232024-10-17T15:07:29.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 1.280523048s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13242024-10-17T15:07:29.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 1.367294636s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
13252024-10-17T15:07:29.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.377074548s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
13262024-10-17T15:07:29.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.011712635s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
13272024-10-17T15:07:29.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 1.05958954s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
13282024-10-17T15:07:29.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.468130845s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
13292024-10-17T15:07:29.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 539.482207ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
13302024-10-17T15:07:30.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 870.210285ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
13312024-10-17T15:07:30.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.227852482s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
13322024-10-17T15:07:30.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.217053891s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
13332024-10-17T15:07:30.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 518.291934ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
13342024-10-17T15:07:30.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 1.143305113s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
13352024-10-17T15:07:30.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.384709207s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
13362024-10-17T15:07:30.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 653.575536ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
13372024-10-17T15:07:30.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.480014673s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
13382024-10-17T15:07:30.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.044164584s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
13392024-10-17T15:07:30.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 555.593779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13402024-10-17T15:07:30.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.406205749s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
13412024-10-17T15:07:30.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 880.519074ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
13422024-10-17T15:07:30.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.231970601s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
13432024-10-17T15:07:30.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.489909118s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
13442024-10-17T15:07:30.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 1.225587689s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
13452024-10-17T15:07:31.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.47189528s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
13462024-10-17T15:07:31.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.249501265s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
13472024-10-17T15:07:31.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 586.030436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13482024-10-17T15:07:31.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.406651706s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
13492024-10-17T15:07:31.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 971.573099ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
13502024-10-17T15:07:31.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 646.094587ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
13512024-10-17T15:07:31.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 852.963944ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
13522024-10-17T15:07:31.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.300484488s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
13532024-10-17T15:07:31.725ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13542024-10-17T15:07:31.726ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
13552024-10-17T15:07:31.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 1.31007741s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
13562024-10-17T15:07:31.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 808.762556ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
13572024-10-17T15:07:31.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 530.050746ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13582024-10-17T15:07:31.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 50.821959ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
13592024-10-17T15:07:31.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.186505796s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
13602024-10-17T15:07:31.979ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13612024-10-17T15:07:31.979ZINFOSledAgent (ServiceManager): Booting oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
13622024-10-17T15:07:32.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 94.359363ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
13632024-10-17T15:07:32.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.179532468s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
13642024-10-17T15:07:32.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 280.607809ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
13652024-10-17T15:07:32.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.015165659s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
13662024-10-17T15:07:32.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.246279812s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
13672024-10-17T15:07:32.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 1.354605743s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
13682024-10-17T15:07:32.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 64.915051ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
13692024-10-17T15:07:32.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.029792142s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
13702024-10-17T15:07:32.306ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13712024-10-17T15:07:32.306ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
13722024-10-17T15:07:32.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 129.205162ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
13732024-10-17T15:07:32.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 1.229819798s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
13742024-10-17T15:07:32.401ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 511.356163ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
13752024-10-17T15:07:32.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 873.558451ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
13762024-10-17T15:07:32.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.278345878s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
13772024-10-17T15:07:32.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 397.18325ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
13782024-10-17T15:07:32.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 239.653195ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
13792024-10-17T15:07:32.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 71.971655ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
13802024-10-17T15:07:32.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.325642415s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
13812024-10-17T15:07:32.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 79.454276ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
13822024-10-17T15:07:32.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.072274189s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
13832024-10-17T15:07:32.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 925.194856ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
13842024-10-17T15:07:32.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 258.962389ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
13852024-10-17T15:07:32.721ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 280.466157ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
13862024-10-17T15:07:32.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 769.70738ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
13872024-10-17T15:07:32.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 505.237245ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
13882024-10-17T15:07:32.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 1.368234404s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
13892024-10-17T15:07:32.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 484.660228ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
13902024-10-17T15:07:33.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 840.269402ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
13912024-10-17T15:07:33.076ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 1.1882469s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
13922024-10-17T15:07:33.194ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 593.810403ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
13932024-10-17T15:07:33.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 892.819572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
13942024-10-17T15:07:33.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.353216603s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
13952024-10-17T15:07:33.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.325105906s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
13962024-10-17T15:07:33.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 612.907154ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
13972024-10-17T15:07:33.370ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13982024-10-17T15:07:33.370ZINFOSledAgent (ServiceManager): Booting oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
13992024-10-17T15:07:33.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.210735506s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
14002024-10-17T15:07:33.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 704.503912ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
14012024-10-17T15:07:33.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 1.116117928s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
14022024-10-17T15:07:33.581ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 772.901106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
14032024-10-17T15:07:33.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.436533643s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
14042024-10-17T15:07:33.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 57.007483ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14052024-10-17T15:07:33.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 1.259611324s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
14062024-10-17T15:07:33.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 1.482069036s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
14072024-10-17T15:07:33.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 81.268981ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14082024-10-17T15:07:33.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 951.257938ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
14092024-10-17T15:07:33.801ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 762.87024ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
14102024-10-17T15:07:33.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 565.044041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
14112024-10-17T15:07:33.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 138.141947ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14122024-10-17T15:07:33.875ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 910.674044ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14132024-10-17T15:07:33.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.416056715s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
14142024-10-17T15:07:33.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.435991672s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
14152024-10-17T15:07:33.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 414.315853ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14162024-10-17T15:07:34.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.348168461s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
14172024-10-17T15:07:34.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 777.011994ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
14182024-10-17T15:07:34.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 519.148636ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
14192024-10-17T15:07:34.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 853.471811ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
14202024-10-17T15:07:34.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 500.553685ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
14212024-10-17T15:07:34.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.198851244s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
14222024-10-17T15:07:34.414ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 1.011472788s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14232024-10-17T15:07:34.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.108509658s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
14242024-10-17T15:07:34.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.229536858s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
14252024-10-17T15:07:34.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 1.417681472s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
14262024-10-17T15:07:34.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.284918764s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
14272024-10-17T15:07:34.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 567.809366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
14282024-10-17T15:07:34.707ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 595.377609ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
14292024-10-17T15:07:34.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 830.440554ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14302024-10-17T15:07:34.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 571.47735ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
14312024-10-17T15:07:34.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 674.26895ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
14322024-10-17T15:07:34.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 1.464372445s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
14332024-10-17T15:07:34.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.220792077s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
14342024-10-17T15:07:35.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.289211084s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
14352024-10-17T15:07:35.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 1.261435791s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
14362024-10-17T15:07:35.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 750.863335ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
14372024-10-17T15:07:35.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 744.934317ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
14382024-10-17T15:07:35.315ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.476105307s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
14392024-10-17T15:07:35.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 990.847733ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
14402024-10-17T15:07:35.395ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 823.010808ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
14412024-10-17T15:07:35.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 811.22707ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
14422024-10-17T15:07:35.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 1.11501843s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14432024-10-17T15:07:35.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.342519619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
14442024-10-17T15:07:35.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 1.428504106s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
14452024-10-17T15:07:35.613ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 854.127927ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
14462024-10-17T15:07:35.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 1.431602404s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14472024-10-17T15:07:35.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 778.428255ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
14482024-10-17T15:07:35.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 885.188069ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
14492024-10-17T15:07:35.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 726.930374ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
14502024-10-17T15:07:35.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 1.409013382s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
14512024-10-17T15:07:36.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.276466957s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
14522024-10-17T15:07:36.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 1.217918212s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
14532024-10-17T15:07:36.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 587.123123ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
14542024-10-17T15:07:36.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614") failed: Property not found. retry in 1.343491183s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
14552024-10-17T15:07:36.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.285734936s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
14562024-10-17T15:07:36.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 1.257966797s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
14572024-10-17T15:07:36.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 971.991267ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
14582024-10-17T15:07:36.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 1.447752602s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
14592024-10-17T15:07:36.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 1.377815197s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
14602024-10-17T15:07:36.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.221819183s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
14612024-10-17T15:07:36.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 1.495896441s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
14622024-10-17T15:07:36.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 625.770084ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14632024-10-17T15:07:36.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 1.363221428s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
14642024-10-17T15:07:36.748ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 967.039378ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
14652024-10-17T15:07:36.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.215415241s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
14662024-10-17T15:07:36.802ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 766.594291ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
14672024-10-17T15:07:36.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1") failed: Property not found. retry in 1.283265112s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
14682024-10-17T15:07:37.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 900.354582ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
14692024-10-17T15:07:37.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 970.992603ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14702024-10-17T15:07:37.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 729.70249ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14712024-10-17T15:07:37.263ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64465
14722024-10-17T15:07:37.266ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1004 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:64465 req_id = 60af1573-8d47-4be9-bbf4-2639b01fdf44 response_code = 200 uri = /baseboard
14732024-10-17T15:07:37.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 858.585346ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
14742024-10-17T15:07:37.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725") failed: Property not found. retry in 1.290254993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
14752024-10-17T15:07:37.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5") failed: Property not found. retry in 1.164595382s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
14762024-10-17T15:07:37.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a") failed: Property not found. retry in 936.41864ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
14772024-10-17T15:07:37.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e") failed: Property not found. retry in 1.292200069s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
14782024-10-17T15:07:37.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8") failed: Property not found. retry in 1.047087027s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
14792024-10-17T15:07:37.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815") failed: Property not found. retry in 620.475806ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
14802024-10-17T15:07:37.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.140424997s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
14812024-10-17T15:07:37.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9") failed: Property not found. retry in 1.346353183s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
14822024-10-17T15:07:37.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78") failed: Property not found. retry in 900.871781ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
14832024-10-17T15:07:37.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 1.176760166s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
14842024-10-17T15:07:37.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4") failed: Property not found. retry in 908.018603ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
14852024-10-17T15:07:37.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 1.154493743s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14862024-10-17T15:07:38.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d") failed: Property not found. retry in 962.425925ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
14872024-10-17T15:07:38.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4") failed: Property not found. retry in 1.09144031s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
14882024-10-17T15:07:38.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f") failed: Property not found. retry in 972.031962ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
14892024-10-17T15:07:38.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 1.157368898s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14902024-10-17T15:07:38.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 1.430326137s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
14912024-10-17T15:07:38.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb") failed: Property not found. retry in 1.002051906s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
14922024-10-17T15:07:39.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 1.133712613s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
14932024-10-17T15:07:39.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 875.102347ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14942024-10-17T15:07:39.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 937.401289ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14952024-10-17T15:07:39.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 659.914353ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
14962024-10-17T15:07:40.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 720.541133ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
14972024-10-17T15:07:40.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 577.502163ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
14982024-10-17T15:07:40.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 788.914811ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
14992024-10-17T15:07:40.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 1.455606062s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
15002024-10-17T15:07:40.773ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 1.26162088s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
15012024-10-17T15:07:40.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 730.809224ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
15022024-10-17T15:07:41.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872") failed: Property not found. retry in 1.287183147s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
15032024-10-17T15:07:41.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b") failed: Property not found. retry in 1.367694344s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
15042024-10-17T15:07:41.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0") failed: Property not found. retry in 1.207007262s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
15052024-10-17T15:07:42.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64") failed: Property not found. retry in 515.738359ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
15062024-10-17T15:07:43.018ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_oximeter_15b75261-ed7f-4731-8d7c-026faffa306d
15072024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_9125336f-8bba-4da8-a415-ccae7d60d01b
15082024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_a25f7721-b451-46d8-90eb-75e61b459dc4
15092024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_external_dns_d8aecb7f-3c94-4690-8de6-10380087d1f0
15102024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_3f0e9f53-0a32-4566-9bce-284027b87815
15112024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_a4097c34-ce96-4b8a-8481-62ca4e7a5dd5
15122024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_74b5cdc0-3a16-484b-9fb3-b7f86509f614
15132024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_clickhouse_f719d96b-fc30-410a-9263-e31911b41872
15142024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_310d3292-985c-40d5-bdf7-ab20ee75d725
15152024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_b794f397-7d13-48f2-bdc7-19ec2afdbebb
15162024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_12909f63-ab2b-4ba9-bf6f-6d95ddb38d9a
15172024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_283be9d0-c6b9-4a8a-b8aa-c7975b11496e
15182024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_b8c2d4d5-8168-42c0-b626-1f4fdd302ca8
15192024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_b24a0bb5-2498-4017-b9e7-384727507bb4
15202024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_9b355ea2-74f1-408f-acfc-0884fd332b64
15212024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_d91e68f0-386e-4018-acb6-b454eb8d2c78
15222024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_external_dns_869dbbb9-e780-4452-80fd-27d18cfc03d9
15232024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_282d1bc9-0cd4-4794-93fb-67f38865892f
15242024-10-17T15:07:43.024ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_b5b2a17f-0793-48a1-96fe-fd8f74634fc1
15252024-10-17T15:07:43.027ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15262024-10-17T15:07:43.031ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
15272024-10-17T15:07:43.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 34917460 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:35325 req_id = 84d6b40d-2d84-4d7c-8d97-5611ef75a882 response_code = 204 uri = /omicron-zones
15282024-10-17T15:07:43.068ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1321
15292024-10-17T15:07:43.068ZINFOSledAgent (RSS): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
15302024-10-17T15:07:43.068ZINFOSledAgent (RSS): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
15312024-10-17T15:07:43.076ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
15322024-10-17T15:07:43.077ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::b]:12221 file = sled-agent/src/rack_setup/service.rs:719
15332024-10-17T15:07:43.113ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.1.174, infra_ip_last: 10.151.1.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.1.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.1.100, rib_priority: None, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:886
15342024-10-17T15:07:43.120ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15352024-10-17T15:07:43.246ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15362024-10-17T15:07:43.366ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15372024-10-17T15:07:43.440ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15382024-10-17T15:07:43.533ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15392024-10-17T15:07:43.677ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15402024-10-17T15:07:43.807ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15412024-10-17T15:07:44.061ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15422024-10-17T15:07:44.584ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15432024-10-17T15:07:45.003ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15442024-10-17T15:07:45.341ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15452024-10-17T15:07:46.085ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15462024-10-17T15:07:47.030ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15472024-10-17T15:07:47.804ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 74.36093784s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
15482024-10-17T15:07:48.080ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15492024-10-17T15:07:48.959ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15502024-10-17T15:07:49.866ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::b]:12221/racks/b4d984d4-c87e-4329-9460-121af5c9b7fe/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
15512024-10-17T15:07:52.427ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "f82082b4-a1dd-4b65-8811-4d0cf2140943", "content-length": "173", "date": "Thu, 17 Oct 2024 15:07:51 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"b4d984d4-c87e-4329-9460-121af5c9b7fe\"", request_id: "f82082b4-a1dd-4b65-8811-4d0cf2140943" } file = sled-agent/src/rack_setup/service.rs:948
15522024-10-17T15:07:55.466ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "7f17563b-4e2d-493a-836a-61e61de6752f", "content-length": "173", "date": "Thu, 17 Oct 2024 15:07:55 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4\"", request_id: "7f17563b-4e2d-493a-836a-61e61de6752f" } file = sled-agent/src/rack_setup/service.rs:948
15532024-10-17T15:07:57.891ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:500 new_generation = 1 old_generation = 1 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
15542024-10-17T15:07:58.028ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45012
15552024-10-17T15:07:58.040ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15562024-10-17T15:07:58.040ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15572024-10-17T15:07:58.040ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:177
15582024-10-17T15:07:58.041ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:177
15592024-10-17T15:07:58.230ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 200858 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45012 req_id = 6416583a-5b05-4205-872e-f81416e4d6ee response_code = 200 uri = /inventory
15602024-10-17T15:07:59.632ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65301
15612024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15622024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15632024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15642024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15652024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15662024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
15672024-10-17T15:07:59.643ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15682024-10-17T15:07:59.643ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 10407 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65301 req_id = 70e14224-9e10-40e2-88fb-0c47d483aa30 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15692024-10-17T15:07:59.644ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:958
15702024-10-17T15:08:00.015ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50869
15712024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15722024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15732024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15742024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15752024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15762024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
15772024-10-17T15:08:00.016ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15782024-10-17T15:08:00.016ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 416 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50869 req_id = cc9d867c-80f8-4290-ae02-8ddfaf6badd2 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15792024-10-17T15:08:00.243ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33162
15802024-10-17T15:08:00.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15812024-10-17T15:08:00.243ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15822024-10-17T15:08:00.243ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/config/omicron-zones.json file = common/src/ledger.rs:177
15832024-10-17T15:08:00.243ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/config/omicron-zones.json file = common/src/ledger.rs:177
15842024-10-17T15:08:00.434ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 190718 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33162 req_id = ea953fa1-1bb9-4029-aad5-f1a519aae912 response_code = 200 uri = /inventory
15852024-10-17T15:08:00.465ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49897
15862024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15872024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15882024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15892024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15902024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15912024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
15922024-10-17T15:08:00.466ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15932024-10-17T15:08:00.466ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 369 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49897 req_id = 2c3dab82-9852-4009-a725-3c537899e5a7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
15942024-10-17T15:08:00.590ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50424
15952024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
15962024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15972024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
15982024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
15992024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
16002024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
16012024-10-17T15:08:00.590ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
16022024-10-17T15:08:00.590ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 319 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50424 req_id = 438ac79e-8879-4238-aebe-650d56840586 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
16032024-10-17T15:08:07.302ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62814
16042024-10-17T15:08:07.303ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:62814 req_id = 04891983-fb7f-4ce2-8261-55f293893412 response_code = 200 uri = /baseboard
16052024-10-17T15:08:11.257ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62519
16062024-10-17T15:08:11.258ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 950 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62519 req_id = b11963c9-665c-4814-80b3-091182e16d1e response_code = 200 uri = /vpc-routes
16072024-10-17T15:08:11.380ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16082024-10-17T15:08:11.380ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 666 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62519 req_id = dc32e629-748c-4705-a7b7-121b7eaf35ee response_code = 204 uri = /eip-gateways
16092024-10-17T15:08:11.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 11497 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:62519 req_id = 878bc3d9-5abe-4133-85d6-e9a3f745088e response_code = 204 uri = /vpc-routes
16102024-10-17T15:08:23.918ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33228
16112024-10-17T15:08:23.918ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 47 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33228 req_id = 24c1d9d0-6281-4cd4-b151-94950a2f7558 response_code = 200 uri = /vpc-routes
16122024-10-17T15:08:24.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34339
16132024-10-17T15:08:24.035ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 833 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34339 req_id = 277fdd42-f29a-41a2-8e5d-072cb45253ab response_code = 200 uri = /v2p
16142024-10-17T15:08:24.116ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49153
16152024-10-17T15:08:24.117ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
16162024-10-17T15:08:24.220ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
16172024-10-17T15:08:24.290ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 174263 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49153 req_id = 24b942a2-295f-4054-919a-fc235ecaf77c response_code = 204 uri = /switch-ports
16182024-10-17T15:08:24.291ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49012
16192024-10-17T15:08:24.291ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16202024-10-17T15:08:24.291ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33228 req_id = bb535685-128b-4e66-81a0-95936f125da1 response_code = 204 uri = /eip-gateways
16212024-10-17T15:08:24.291ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49012 req_id = 7b473cf5-cf3f-4a73-baa0-020f7af051d0 response_code = 200 uri = /vpc-routes
16222024-10-17T15:08:24.292ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1391 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49153 req_id = 63864755-0418-4ec1-a0ed-37b1035580ab response_code = 200 uri = /network-bootstore-config
16232024-10-17T15:08:24.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 374 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33228 req_id = 59adcd6f-462f-481a-ac4c-5a124e5fd723 response_code = 204 uri = /vpc-routes
16242024-10-17T15:08:24.307ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47742
16252024-10-17T15:08:24.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47742 req_id = de1c92fe-e971-4c49-ae2c-388dbb35c0a3 response_code = 200 uri = /vpc-routes
16262024-10-17T15:08:24.329ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16272024-10-17T15:08:24.329ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47742 req_id = 3972cb63-3d67-4e04-86a9-008fd902479c response_code = 204 uri = /eip-gateways
16282024-10-17T15:08:24.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 246 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:47742 req_id = 3f4ce9ea-1ec4-4c72-9484-16e44522fd90 response_code = 204 uri = /vpc-routes
16292024-10-17T15:08:24.397ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16302024-10-17T15:08:24.397ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49012 req_id = 6c81124b-8715-4c18-a70f-a97b6dc0b0c0 response_code = 204 uri = /eip-gateways
16312024-10-17T15:08:24.397ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58945
16322024-10-17T15:08:24.397ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
16332024-10-17T15:08:24.434ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
16342024-10-17T15:08:24.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88768 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58945 req_id = 3c0cafc2-d355-477e-a4d0-400566b1d4f5 response_code = 204 uri = /switch-ports
16352024-10-17T15:08:24.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56445
16362024-10-17T15:08:24.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 321 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49012 req_id = 907f4be6-b31f-4e20-bd54-d7d21744d5e2 response_code = 204 uri = /vpc-routes
16372024-10-17T15:08:24.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 244 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58945 req_id = 32c38a8d-489c-4ca8-a915-865a013b14e6 response_code = 200 uri = /network-bootstore-config
16382024-10-17T15:08:24.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 221 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56445 req_id = 8d3141c0-e222-4fc9-b125-73aea9256fa4 response_code = 200 uri = /v2p
16392024-10-17T15:08:24.526ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-buskin-i86pc
16402024-10-17T15:08:24.526ZINFOSledAgent (bootstore): Writing ledger to /pool/int/34423522-8473-4e67-b21d-dccdbba2931c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
16412024-10-17T15:08:24.527ZINFOSledAgent (bootstore): Writing ledger to /pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-buskin-i86pc
16422024-10-17T15:08:24.528ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-buskin-i86pc
16432024-10-17T15:08:24.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2389 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:58945 req_id = f15a787d-4d36-4525-a616-8e4b80d5d05b response_code = 204 uri = /network-bootstore-config
16442024-10-17T15:08:24.552ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63698
16452024-10-17T15:08:24.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63698 req_id = c1e5f81b-b5af-4432-beb3-b845a9e15e35 response_code = 200 uri = /v2p
16462024-10-17T15:08:24.652ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38585
16472024-10-17T15:08:24.652ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
16482024-10-17T15:08:24.688ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
16492024-10-17T15:08:24.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88907 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:38585 req_id = b5d5b497-e87f-4faf-b3d3-1caf75d8ed69 response_code = 204 uri = /switch-ports
16502024-10-17T15:08:24.741ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38585 req_id = dc6ae396-e8cb-4123-a11c-df11128fd12e response_code = 200 uri = /network-bootstore-config
16512024-10-17T15:08:37.343ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39142
16522024-10-17T15:08:37.343ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 42 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:39142 req_id = 26f26d21-7fb1-4c58-955c-d27c66d000fa response_code = 200 uri = /baseboard
16532024-10-17T15:08:54.078ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40143
16542024-10-17T15:08:54.079ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40143 req_id = 735db844-aaa3-4872-9a87-1ac4587f60bb response_code = 200 uri = /v2p
16552024-10-17T15:08:54.324ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43813
16562024-10-17T15:08:54.324ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43813 req_id = 4eb94b90-f8eb-4ef9-a412-87d77489b9ff response_code = 200 uri = /vpc-routes
16572024-10-17T15:08:54.392ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16582024-10-17T15:08:54.392ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43813 req_id = 3fa623ac-4b3c-43cb-b948-a99bbcc74f7f response_code = 204 uri = /eip-gateways
16592024-10-17T15:08:54.396ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:36736
16602024-10-17T15:08:54.396ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:36736 req_id = 970be221-e15c-4981-9a18-5056a2ca250e response_code = 200 uri = /vpc-routes
16612024-10-17T15:08:54.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43813 req_id = cfb0f1a7-4617-4041-9d18-9dff612ee273 response_code = 204 uri = /vpc-routes
16622024-10-17T15:08:54.403ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56109
16632024-10-17T15:08:54.403ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56109 req_id = e098cc2d-2c47-4a67-8a69-07cc52b0261d response_code = 200 uri = /vpc-routes
16642024-10-17T15:08:54.463ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16652024-10-17T15:08:54.464ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36736 req_id = 40c7febe-78ee-4c24-80a8-991233dc0d92 response_code = 204 uri = /eip-gateways
16662024-10-17T15:08:54.470ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16672024-10-17T15:08:54.470ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56109 req_id = 8b79aa13-648b-4dcb-96de-f0f46de0a047 response_code = 204 uri = /eip-gateways
16682024-10-17T15:08:54.472ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:36736 req_id = 71fbbf2c-69ae-47a8-9661-54f5fadf7064 response_code = 204 uri = /vpc-routes
16692024-10-17T15:08:54.542ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65006
16702024-10-17T15:08:54.542ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65006 req_id = 41410fa0-f459-448e-81e4-7501529e1144 response_code = 200 uri = /v2p
16712024-10-17T15:08:54.546ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56109 req_id = 04c7a362-7d04-4bb0-ba8d-c6ad9f902ab6 response_code = 204 uri = /vpc-routes
16722024-10-17T15:08:54.557ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34856
16732024-10-17T15:08:54.557ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
16742024-10-17T15:08:54.592ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
16752024-10-17T15:08:54.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89398 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34856 req_id = 9eb2f5fc-2613-43ca-993e-5bd4904a8421 response_code = 204 uri = /switch-ports
16762024-10-17T15:08:54.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35429
16772024-10-17T15:08:54.651ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
16782024-10-17T15:08:54.656ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8773 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34856 req_id = 2caaab32-e473-4126-b470-5862fac19969 response_code = 200 uri = /network-bootstore-config
16792024-10-17T15:08:54.660ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64829
16802024-10-17T15:08:54.690ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
16812024-10-17T15:08:54.707ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52109
16822024-10-17T15:08:54.707ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52109 req_id = a027d5be-0155-478c-8ba1-0a10e17c31a3 response_code = 200 uri = /v2p
16832024-10-17T15:08:54.742ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
16842024-10-17T15:08:54.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91424 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35429 req_id = 2ecffeff-00a3-49e1-b815-58dd8b362861 response_code = 204 uri = /switch-ports
16852024-10-17T15:08:54.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 771 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35429 req_id = 683d13b4-1b49-4854-ab33-7d0b5c5562f1 response_code = 200 uri = /network-bootstore-config
16862024-10-17T15:08:54.777ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
16872024-10-17T15:08:54.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 171068 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:64829 req_id = 42162c15-5bfc-4aa7-b136-e195f8a2f827 response_code = 204 uri = /switch-ports
16882024-10-17T15:08:54.832ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64829 req_id = 1234f42a-f622-48a8-a459-d0a7715f82ef response_code = 200 uri = /network-bootstore-config
16892024-10-17T15:09:07.376ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61307
16902024-10-17T15:09:07.377ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:61307 req_id = 9c698fc5-d508-48c9-83fa-ea95d18df64a response_code = 200 uri = /baseboard
16912024-10-17T15:09:23.993ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61182
16922024-10-17T15:09:23.994ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 207 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61182 req_id = 26aa244f-5df9-4857-8d11-a257c89424aa response_code = 200 uri = /v2p
16932024-10-17T15:09:23.998ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54063
16942024-10-17T15:09:23.998ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54063 req_id = 2045188e-059a-4359-acb2-2282e4e9d7c1 response_code = 200 uri = /vpc-routes
16952024-10-17T15:09:24.039ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
16962024-10-17T15:09:24.039ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54063 req_id = 12d9d236-9ff0-4542-a5af-6f6791c0b216 response_code = 204 uri = /eip-gateways
16972024-10-17T15:09:24.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 198 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:54063 req_id = dd09c88b-78ea-47d5-a830-05f5738cc6d6 response_code = 204 uri = /vpc-routes
16982024-10-17T15:09:24.097ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42394
16992024-10-17T15:09:24.097ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17002024-10-17T15:09:24.145ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17012024-10-17T15:09:24.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124327 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42394 req_id = 501e1d2f-4ff9-4c46-af48-d7546b576ac2 response_code = 204 uri = /switch-ports
17022024-10-17T15:09:24.221ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42394 req_id = bc09c597-2df1-4ca1-8997-29778bd452bb response_code = 200 uri = /network-bootstore-config
17032024-10-17T15:09:24.245ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41447
17042024-10-17T15:09:24.246ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41447 req_id = d1536ffc-0be6-4ad5-ad68-1f48ee1ef8b4 response_code = 200 uri = /vpc-routes
17052024-10-17T15:09:24.279ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17062024-10-17T15:09:24.279ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41447 req_id = 42d119c2-8ed6-489b-81f5-fc06c6f7cc0d response_code = 204 uri = /eip-gateways
17072024-10-17T15:09:24.320ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35372
17082024-10-17T15:09:24.320ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35372 req_id = 6953df95-5478-47fd-829a-fe2ccd4920b1 response_code = 200 uri = /vpc-routes
17092024-10-17T15:09:24.345ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40029
17102024-10-17T15:09:24.345ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40029 req_id = 8ccb2605-bf97-4117-9bb4-99e67af6a3c2 response_code = 200 uri = /v2p
17112024-10-17T15:09:24.364ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17122024-10-17T15:09:24.364ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35372 req_id = 66ac79e1-82ac-48be-bef4-c563d2c05ffe response_code = 204 uri = /eip-gateways
17132024-10-17T15:09:24.382ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41447 req_id = 42a71e5b-5dcf-49aa-a032-22402f360610 response_code = 204 uri = /vpc-routes
17142024-10-17T15:09:24.392ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46786
17152024-10-17T15:09:24.392ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17162024-10-17T15:09:24.430ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17172024-10-17T15:09:24.485ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93243 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46786 req_id = 88d9f1da-91cc-4a81-bacb-3fdce4824025 response_code = 204 uri = /switch-ports
17182024-10-17T15:09:24.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53410
17192024-10-17T15:09:24.486ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58107
17202024-10-17T15:09:24.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 335 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35372 req_id = 1c665bd2-faff-4afb-9528-1473b39eb3f6 response_code = 204 uri = /vpc-routes
17212024-10-17T15:09:24.486ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17222024-10-17T15:09:24.486ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 324 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46786 req_id = c672bbe9-00b0-4c5d-9e17-abb798f515e9 response_code = 200 uri = /network-bootstore-config
17232024-10-17T15:09:24.491ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4863 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53410 req_id = 18c9da49-17a1-4faa-9a47-140a0ea7699a response_code = 200 uri = /v2p
17242024-10-17T15:09:24.521ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17252024-10-17T15:09:24.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90458 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58107 req_id = dc4b9374-7dc4-4123-b44d-b0400b99dabe response_code = 204 uri = /switch-ports
17262024-10-17T15:09:24.577ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58107 req_id = 9ab7f6ad-76cf-4942-b6f1-638bfc8f940a response_code = 200 uri = /network-bootstore-config
17272024-10-17T15:09:31.776ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
17282024-10-17T15:09:32.272ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:44276 remote_addr = [fd00:1122:3344:101::d]:40319 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
17292024-10-17T15:09:32.308ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 34804 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = ba15538d-190b-4c60-a469-206dcae1f751 response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
17302024-10-17T15:09:37.410ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45639
17312024-10-17T15:09:37.410ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45639 req_id = 53f65223-b8b4-4bb2-9d22-7a2e1dfc1937 response_code = 200 uri = /baseboard
17322024-10-17T15:09:53.995ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51003
17332024-10-17T15:09:53.995ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51003 req_id = 3caa222d-e3b2-4728-8702-40fcabf9f5cd response_code = 200 uri = /vpc-routes
17342024-10-17T15:09:54.057ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17352024-10-17T15:09:54.057ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51003 req_id = 0283eb8c-0a7f-440f-9e6a-eaeb5104733f response_code = 204 uri = /eip-gateways
17362024-10-17T15:09:54.240ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59048
17372024-10-17T15:09:54.240ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 738 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59048 req_id = 2a3027c3-dd28-4b58-97e7-0223b04a9451 response_code = 200 uri = /v2p
17382024-10-17T15:09:54.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 275 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51003 req_id = 50664f34-4148-4c52-a57d-cadd63b62310 response_code = 204 uri = /vpc-routes
17392024-10-17T15:09:54.448ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42645
17402024-10-17T15:09:54.448ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17412024-10-17T15:09:54.488ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17422024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104391 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42645 req_id = 2331776b-df47-4af0-9b7f-a9fc5fe94f5c response_code = 204 uri = /switch-ports
17432024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56454
17442024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57650
17452024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59271
17462024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59271 req_id = 97d2d23f-1b31-4594-af51-68a3816c6f8a response_code = 200 uri = /vpc-routes
17472024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57650 req_id = 5af34678-f46a-40a6-9016-92b601927de1 response_code = 200 uri = /vpc-routes
17482024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 273 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56454 req_id = 4e735207-55b5-4755-bc8f-28339cf1ea25 response_code = 200 uri = /v2p
17492024-10-17T15:09:54.553ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 279 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42645 req_id = dd03492d-8b0f-44ed-bb85-7af3d91fc4d1 response_code = 200 uri = /network-bootstore-config
17502024-10-17T15:09:54.669ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17512024-10-17T15:09:54.669ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57650 req_id = 49db59d2-6994-44f5-8596-9622adab1e83 response_code = 204 uri = /eip-gateways
17522024-10-17T15:09:54.669ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17532024-10-17T15:09:54.669ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59271 req_id = f8e19dbe-a2f0-4d09-8c03-5023b4882519 response_code = 204 uri = /eip-gateways
17542024-10-17T15:09:54.670ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63235
17552024-10-17T15:09:54.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63235 req_id = d535c339-3762-44c6-8cfd-4f16b52f6fc0 response_code = 200 uri = /v2p
17562024-10-17T15:09:54.677ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1017 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:59271 req_id = 929c63d6-bdfc-4371-b188-e05d3228416e response_code = 204 uri = /vpc-routes
17572024-10-17T15:09:54.702ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43866
17582024-10-17T15:09:54.702ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17592024-10-17T15:09:54.740ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17602024-10-17T15:09:54.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94391 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43866 req_id = 7f7c6ccd-d5d6-423e-b992-9b0e2f549074 response_code = 204 uri = /switch-ports
17612024-10-17T15:09:54.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 224 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57650 req_id = 514af000-d3fb-496c-8905-141f04063924 response_code = 204 uri = /vpc-routes
17622024-10-17T15:09:54.797ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43866 req_id = 3c5057ee-3abd-4515-91fb-2440265f8635 response_code = 200 uri = /network-bootstore-config
17632024-10-17T15:09:54.803ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45003
17642024-10-17T15:09:54.804ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17652024-10-17T15:09:54.843ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17662024-10-17T15:09:54.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94238 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:45003 req_id = 03ff6c7d-2fac-4868-814a-6d1e55a62c71 response_code = 204 uri = /switch-ports
17672024-10-17T15:09:54.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45003 req_id = 422be62d-4179-4072-8add-7e1c1d34da23 response_code = 200 uri = /network-bootstore-config
17682024-10-17T15:10:02.277ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6225 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = 38f6fa39-a366-49ad-97e8-92ecd56a63df response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
17692024-10-17T15:10:07.444ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45819
17702024-10-17T15:10:07.444ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 62 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:45819 req_id = 027f959c-a484-4c67-bc2b-e3d3d4cd658b response_code = 200 uri = /baseboard
17712024-10-17T15:10:23.872ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43776
17722024-10-17T15:10:23.872ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43776 req_id = 0ae6fd76-aadc-44f9-b8f8-5b3ebaf68c92 response_code = 200 uri = /v2p
17732024-10-17T15:10:24.015ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53782
17742024-10-17T15:10:24.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 68 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53782 req_id = 6442b9f5-53e3-47f4-b14d-4155dd88e4fa response_code = 200 uri = /vpc-routes
17752024-10-17T15:10:24.061ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17762024-10-17T15:10:24.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53782 req_id = cd67ed9c-515a-4e55-8ccd-eec2e7b29b8e response_code = 204 uri = /eip-gateways
17772024-10-17T15:10:24.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53782 req_id = 2699f5ee-6352-4a29-86f0-f53d74f5f51d response_code = 204 uri = /vpc-routes
17782024-10-17T15:10:24.238ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41245
17792024-10-17T15:10:24.238ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41245 req_id = 3cf4cce5-b58e-4cc8-8985-9046dd685371 response_code = 200 uri = /vpc-routes
17802024-10-17T15:10:24.256ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50005
17812024-10-17T15:10:24.256ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17822024-10-17T15:10:24.300ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17832024-10-17T15:10:24.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108667 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50005 req_id = 34b1c2f6-a042-4ae3-8b60-aa9ee365f2cf response_code = 204 uri = /switch-ports
17842024-10-17T15:10:24.365ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60529
17852024-10-17T15:10:24.365ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
17862024-10-17T15:10:24.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41245 req_id = 7ea47371-37fa-4fe2-af10-8d93c891fd3e response_code = 204 uri = /eip-gateways
17872024-10-17T15:10:24.365ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50232
17882024-10-17T15:10:24.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60529 req_id = c9d598dc-84c8-4ccc-aa67-4873e0052b74 response_code = 200 uri = /v2p
17892024-10-17T15:10:24.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 298 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50005 req_id = 3310dd26-e1ea-438b-b657-21504788efa3 response_code = 200 uri = /network-bootstore-config
17902024-10-17T15:10:24.365ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
17912024-10-17T15:10:24.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41245 req_id = 9d13c2ac-5603-471a-8ac5-b080cb423d76 response_code = 204 uri = /vpc-routes
17922024-10-17T15:10:24.406ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
17932024-10-17T15:10:24.414ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61320
17942024-10-17T15:10:24.414ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61320 req_id = 5b77c2b0-54a1-42e4-ae1a-37dc87ad4fe5 response_code = 200 uri = /vpc-routes
17952024-10-17T15:10:24.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97966 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50232 req_id = ddf20a6d-b58e-4ea9-9a2e-60ff3bdcc466 response_code = 204 uri = /switch-ports
17962024-10-17T15:10:24.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50232 req_id = a0d8226c-af58-4e52-b853-d7d87d62ab4f response_code = 200 uri = /network-bootstore-config
17972024-10-17T15:10:24.480ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52038
17982024-10-17T15:10:24.480ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52038 req_id = b4e6f0c1-9c5b-4a91-b38e-51b8f52b7284 response_code = 200 uri = /v2p
17992024-10-17T15:10:24.525ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18002024-10-17T15:10:24.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61320 req_id = f2bc49bf-fcac-4b65-9f2f-d6b7f250dc5b response_code = 204 uri = /eip-gateways
18012024-10-17T15:10:24.552ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39737
18022024-10-17T15:10:24.552ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18032024-10-17T15:10:24.588ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18042024-10-17T15:10:24.645ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93219 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39737 req_id = f7ca350d-5ecd-4d3a-aaef-9fed5cc52007 response_code = 204 uri = /switch-ports
18052024-10-17T15:10:24.649ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3532 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39737 req_id = 71a440b5-fd37-412f-88bb-6435e8988a48 response_code = 200 uri = /network-bootstore-config
18062024-10-17T15:10:24.649ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3648 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61320 req_id = cc783281-61d4-46c9-a68b-d159dd2bd6b9 response_code = 204 uri = /vpc-routes
18072024-10-17T15:10:32.279ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7896 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = f060b062-8dda-4a9b-bf4a-0210ba82d93d response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
18082024-10-17T15:10:37.477ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37650
18092024-10-17T15:10:37.478ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 59 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:37650 req_id = fa5b71a7-de94-4131-a7cd-4d5e22738a4c response_code = 200 uri = /baseboard
18102024-10-17T15:10:54.318ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55097
18112024-10-17T15:10:54.318ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18122024-10-17T15:10:54.369ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18132024-10-17T15:10:54.431ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 112838 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55097 req_id = 1ffa13f3-d12f-4775-bd0e-07e9961828ae response_code = 204 uri = /switch-ports
18142024-10-17T15:10:54.431ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44433
18152024-10-17T15:10:54.432ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50739
18162024-10-17T15:10:54.432ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55097 req_id = 08f4b119-2872-4665-8f99-2586649af792 response_code = 200 uri = /network-bootstore-config
18172024-10-17T15:10:54.432ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44433 req_id = ba30542e-c716-41f5-bcd7-0acb0a7f34d5 response_code = 200 uri = /vpc-routes
18182024-10-17T15:10:54.432ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50739 req_id = 86559f01-22b6-425d-bd0d-14e00ab62f40 response_code = 200 uri = /v2p
18192024-10-17T15:10:54.435ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33434
18202024-10-17T15:10:54.435ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 33 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33434 req_id = 84bdabd2-3a30-4c38-9041-89ef6cf3463a response_code = 200 uri = /vpc-routes
18212024-10-17T15:10:54.498ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18222024-10-17T15:10:54.498ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44433 req_id = 52c0c006-ad55-4f9c-bfc3-90a0bd4b7db1 response_code = 204 uri = /eip-gateways
18232024-10-17T15:10:54.510ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 288 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44433 req_id = c12a90de-0eff-4dcd-8f29-04537034e044 response_code = 204 uri = /vpc-routes
18242024-10-17T15:10:54.530ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18252024-10-17T15:10:54.530ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33434 req_id = 3a6d120c-73de-4ae4-9ece-07e41f9ca7aa response_code = 204 uri = /eip-gateways
18262024-10-17T15:10:54.539ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 238 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33434 req_id = e9799489-aa57-4299-b64d-d5c876716829 response_code = 204 uri = /vpc-routes
18272024-10-17T15:10:54.552ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48753
18282024-10-17T15:10:54.553ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18292024-10-17T15:10:54.594ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18302024-10-17T15:10:54.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 111962 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48753 req_id = ec85edc3-e702-49a3-ae3c-efa14886b279 response_code = 204 uri = /switch-ports
18312024-10-17T15:10:54.665ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40285
18322024-10-17T15:10:54.665ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35552
18332024-10-17T15:10:54.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35552 req_id = 014c51c5-2760-4124-bda4-ab66ad1be041 response_code = 200 uri = /v2p
18342024-10-17T15:10:54.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 262 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48753 req_id = 29f70d5e-f4a6-49a3-bb7f-c6b7d9070c63 response_code = 200 uri = /network-bootstore-config
18352024-10-17T15:10:54.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40285 req_id = 778e4eda-7532-404e-9d77-97ac6afb49a4 response_code = 200 uri = /vpc-routes
18362024-10-17T15:10:54.695ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58344
18372024-10-17T15:10:54.695ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18382024-10-17T15:10:54.735ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18392024-10-17T15:10:54.792ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97393 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58344 req_id = 4c8cae8b-84a5-4f46-afa9-9e027ce13dff response_code = 204 uri = /switch-ports
18402024-10-17T15:10:54.792ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18412024-10-17T15:10:54.792ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40285 req_id = 6a73125c-b565-4050-b121-e094b9df4301 response_code = 204 uri = /eip-gateways
18422024-10-17T15:10:54.793ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58344 req_id = 7ff5df81-599b-4048-a946-3831cfd17a4b response_code = 200 uri = /network-bootstore-config
18432024-10-17T15:10:54.824ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35751
18442024-10-17T15:10:54.824ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35751 req_id = cc03c8a9-a78f-4b49-93cf-69cf9efa8ed0 response_code = 200 uri = /v2p
18452024-10-17T15:10:54.831ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 191 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:40285 req_id = 7ec32d3c-f8b6-4639-a510-e097767e24b9 response_code = 204 uri = /vpc-routes
18462024-10-17T15:11:02.278ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6533 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = 5f7fd0ff-ed78-4f33-8947-6b216d85624d response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
18472024-10-17T15:11:07.512ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47389
18482024-10-17T15:11:07.513ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 53 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:47389 req_id = 5af19ae2-b267-42e6-811c-b5bfd9ace90f response_code = 200 uri = /baseboard
18492024-10-17T15:11:23.856ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44744
18502024-10-17T15:11:23.856ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44744 req_id = 111f8c7c-30a8-4164-82e9-37aedc8890e2 response_code = 200 uri = /vpc-routes
18512024-10-17T15:11:23.873ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18522024-10-17T15:11:23.873ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44744 req_id = d6a333f6-7614-4604-bc7f-5b55b629b60a response_code = 204 uri = /eip-gateways
18532024-10-17T15:11:23.881ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44744 req_id = 35e2d0c4-bfab-47c0-b6ff-84d7c9ac3551 response_code = 204 uri = /vpc-routes
18542024-10-17T15:11:23.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42645
18552024-10-17T15:11:23.996ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18562024-10-17T15:11:24.034ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18572024-10-17T15:11:24.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107475 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:42645 req_id = 55166faa-29c3-45ba-a0a2-926dbaaebb35 response_code = 204 uri = /switch-ports
18582024-10-17T15:11:24.103ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37774
18592024-10-17T15:11:24.104ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 227 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42645 req_id = ebce7681-6c91-4bc9-b313-cc9aff7cae89 response_code = 200 uri = /network-bootstore-config
18602024-10-17T15:11:24.104ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 225 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37774 req_id = 1d23c0de-c298-46df-bdcc-cc877e92e9ca response_code = 200 uri = /v2p
18612024-10-17T15:11:24.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37984
18622024-10-17T15:11:24.148ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37984 req_id = c77450e8-7c7f-437b-97d8-7d50dccd48fb response_code = 200 uri = /vpc-routes
18632024-10-17T15:11:24.219ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18642024-10-17T15:11:24.219ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37984 req_id = ee021b1f-806d-403f-a4a2-fc120122dee1 response_code = 204 uri = /eip-gateways
18652024-10-17T15:11:24.230ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 226 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37984 req_id = e605ae85-5c8f-42c5-9d05-568c7927727d response_code = 204 uri = /vpc-routes
18662024-10-17T15:11:24.305ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64695
18672024-10-17T15:11:24.306ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18682024-10-17T15:11:24.348ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18692024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104135 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:64695 req_id = 8e61d012-01be-4892-8ee5-64da8a7af966 response_code = 204 uri = /switch-ports
18702024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65017
18712024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55414
18722024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38973
18732024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 204 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38973 req_id = 0b6947ef-47a1-4a14-96b7-919ece5f9814 response_code = 200 uri = /v2p
18742024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 237 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64695 req_id = 0bab4acd-7ac1-451d-998b-9f0a4cffc8d6 response_code = 200 uri = /network-bootstore-config
18752024-10-17T15:11:24.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65017 req_id = 3e82610b-0099-43aa-9884-f30ff474e57c response_code = 200 uri = /v2p
18762024-10-17T15:11:24.411ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55414 req_id = 38a6dc89-7318-4922-9dba-34c0cee66fac response_code = 200 uri = /vpc-routes
18772024-10-17T15:11:24.429ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38636
18782024-10-17T15:11:24.429ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18792024-10-17T15:11:24.469ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18802024-10-17T15:11:24.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96380 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:38636 req_id = 162b35b2-769c-4b6b-a77e-a5550bc4a85b response_code = 204 uri = /switch-ports
18812024-10-17T15:11:24.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38636 req_id = dd64a4cb-dc4c-43a3-9c6e-829c028afde1 response_code = 200 uri = /network-bootstore-config
18822024-10-17T15:11:24.526ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18832024-10-17T15:11:24.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55414 req_id = 9c557a58-eb3c-47ad-a396-371a72bb98d7 response_code = 204 uri = /eip-gateways
18842024-10-17T15:11:24.535ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 242 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55414 req_id = d118502b-d3b4-4cde-bdc2-a5f9a4e326db response_code = 204 uri = /vpc-routes
18852024-10-17T15:11:32.281ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6680 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = 0c7de6ab-c840-4adf-a656-295c49192aef response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
18862024-10-17T15:11:37.552ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59781
18872024-10-17T15:11:37.552ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:59781 req_id = 48d57170-c96d-4e46-ac69-36292653abd7 response_code = 200 uri = /baseboard
18882024-10-17T15:11:54.091ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44278
18892024-10-17T15:11:54.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44278 req_id = efe2869d-a20c-4e43-ad39-a30fae451198 response_code = 200 uri = /v2p
18902024-10-17T15:11:54.210ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36708
18912024-10-17T15:11:54.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36708 req_id = 1eee95c6-1c90-465a-9ebc-153b02dda0e1 response_code = 200 uri = /vpc-routes
18922024-10-17T15:11:54.248ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18932024-10-17T15:11:54.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36708 req_id = d08750e6-0716-44ea-9ba3-5e17eb32b9b0 response_code = 204 uri = /eip-gateways
18942024-10-17T15:11:54.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 317 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:36708 req_id = affbbadd-c975-4bbd-995b-c732f2d7128a response_code = 204 uri = /vpc-routes
18952024-10-17T15:11:54.374ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37407
18962024-10-17T15:11:54.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37407 req_id = 9cb80c0f-06a9-4fd2-9e89-f27a43e83576 response_code = 200 uri = /vpc-routes
18972024-10-17T15:11:54.428ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
18982024-10-17T15:11:54.428ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37407 req_id = 1bfa6b57-20af-4a68-9ff3-b21be5488654 response_code = 204 uri = /eip-gateways
18992024-10-17T15:11:54.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 302 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37407 req_id = 3d66f30a-e6cd-4ab4-906b-0306170124ef response_code = 204 uri = /vpc-routes
19002024-10-17T15:11:54.476ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40968
19012024-10-17T15:11:54.477ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19022024-10-17T15:11:54.527ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19032024-10-17T15:11:54.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129828 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40968 req_id = 412bde08-fb70-49aa-8e66-590e3fed39b7 response_code = 204 uri = /switch-ports
19042024-10-17T15:11:54.607ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57622
19052024-10-17T15:11:54.607ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19062024-10-17T15:11:54.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4086 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40968 req_id = 47d8bd15-a09e-45d5-ab4e-72cdfbeb66db response_code = 200 uri = /network-bootstore-config
19072024-10-17T15:11:54.627ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34254
19082024-10-17T15:11:54.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34254 req_id = bcb73dd5-df87-4f4c-b180-a062816ddcba response_code = 200 uri = /v2p
19092024-10-17T15:11:54.650ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19102024-10-17T15:11:54.675ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56233
19112024-10-17T15:11:54.675ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56233 req_id = 59e75852-b8a2-4f1c-82b8-870403c3c2d7 response_code = 200 uri = /vpc-routes
19122024-10-17T15:11:54.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109265 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57622 req_id = e58e239a-f6c1-4443-b1ff-8e5d5087a873 response_code = 204 uri = /switch-ports
19132024-10-17T15:11:54.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57622 req_id = 85a91b53-504e-426f-b91b-a76a7d691289 response_code = 200 uri = /network-bootstore-config
19142024-10-17T15:11:54.727ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
19152024-10-17T15:11:54.727ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56233 req_id = 08df4356-84ad-482b-8df5-c6244d59b257 response_code = 204 uri = /eip-gateways
19162024-10-17T15:11:54.737ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48979
19172024-10-17T15:11:54.738ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48979 req_id = fad576a2-eb88-4b34-a142-af01ca7c2593 response_code = 200 uri = /v2p
19182024-10-17T15:11:54.740ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56883
19192024-10-17T15:11:54.740ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19202024-10-17T15:11:54.773ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19212024-10-17T15:11:54.825ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84899 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56883 req_id = 5fb28b17-e87a-41ce-b485-370a0ccc1456 response_code = 204 uri = /switch-ports
19222024-10-17T15:11:54.825ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56883 req_id = f158be35-a4fe-4aaa-a7ba-9db9a6e4ce75 response_code = 200 uri = /network-bootstore-config
19232024-10-17T15:11:54.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 481 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56233 req_id = 6d1df0ca-e13c-4fb9-b5ff-9868bd78e418 response_code = 204 uri = /vpc-routes
19242024-10-17T15:12:02.420ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6552 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = 983f9dc3-ba08-4ef8-8c0f-623a1a14f3e0 response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
19252024-10-17T15:12:07.587ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57012
19262024-10-17T15:12:07.587ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:57012 req_id = 6e543d5c-1fff-47b8-8d8c-2ecb23b63a30 response_code = 200 uri = /baseboard
19272024-10-17T15:12:23.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52979
19282024-10-17T15:12:23.825ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52979 req_id = caa0d37c-b40d-4b9c-ae57-e0a01196941a response_code = 200 uri = /vpc-routes
19292024-10-17T15:12:23.840ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
19302024-10-17T15:12:23.840ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:52979 req_id = d360e98e-9cc1-4f33-af6e-138e4310f50e response_code = 204 uri = /eip-gateways
19312024-10-17T15:12:23.848ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 177 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:52979 req_id = 7018ddf1-0377-429f-91bc-a7b4769f8d1e response_code = 204 uri = /vpc-routes
19322024-10-17T15:12:24.049ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60176
19332024-10-17T15:12:24.050ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 168 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60176 req_id = f2c266f0-36f5-4f8e-9226-6da71b663f1a response_code = 200 uri = /v2p
19342024-10-17T15:12:24.169ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58146
19352024-10-17T15:12:24.169ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19362024-10-17T15:12:24.221ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19372024-10-17T15:12:24.302ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133068 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:58146 req_id = 6808b8c6-ca35-4148-9424-022f1eeec313 response_code = 204 uri = /switch-ports
19382024-10-17T15:12:24.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37440
19392024-10-17T15:12:24.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37440 req_id = b73e8c95-91a5-46c7-9a4b-e0aafe93d672 response_code = 200 uri = /vpc-routes
19402024-10-17T15:12:24.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58146 req_id = 385c533b-da9c-47b5-aad3-393ee306bc4f response_code = 200 uri = /network-bootstore-config
19412024-10-17T15:12:24.350ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
19422024-10-17T15:12:24.350ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37440 req_id = 7215e198-b5b0-4060-a107-aa9abf9a1ffc response_code = 204 uri = /eip-gateways
19432024-10-17T15:12:24.366ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62722
19442024-10-17T15:12:24.366ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 49 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62722 req_id = ff83190d-cfa5-437e-a814-636578278440 response_code = 200 uri = /vpc-routes
19452024-10-17T15:12:24.406ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
19462024-10-17T15:12:24.406ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62722 req_id = 486b2f1a-182d-4df3-8839-c07c57345a60 response_code = 204 uri = /eip-gateways
19472024-10-17T15:12:24.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 334 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62722 req_id = 0a596095-6123-4cb8-b3f6-2b83aa65db38 response_code = 204 uri = /vpc-routes
19482024-10-17T15:12:24.442ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53918
19492024-10-17T15:12:24.442ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53918 req_id = 0f96fa0e-733d-41fb-bc08-65e03369b388 response_code = 200 uri = /v2p
19502024-10-17T15:12:24.452ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37440 req_id = b1ff2d3d-7446-4e8b-92ef-22bc57da981c response_code = 204 uri = /vpc-routes
19512024-10-17T15:12:24.455ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62746
19522024-10-17T15:12:24.455ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62746 req_id = b27a56b7-3ae5-4988-ad45-315f5cd26e30 response_code = 200 uri = /v2p
19532024-10-17T15:12:24.500ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41816
19542024-10-17T15:12:24.500ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19552024-10-17T15:12:24.538ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19562024-10-17T15:12:24.600ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100077 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41816 req_id = d74f8716-6e6b-4f18-a23a-319f2b4dfaf2 response_code = 204 uri = /switch-ports
19572024-10-17T15:12:24.600ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52775
19582024-10-17T15:12:24.601ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19592024-10-17T15:12:24.635ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19602024-10-17T15:12:24.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90491 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52775 req_id = e6f9df28-baf2-4296-93c9-a38b4ea6841c response_code = 204 uri = /switch-ports
19612024-10-17T15:12:24.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52775 req_id = b112e009-f3e4-41d4-926d-bd2b4625eaec response_code = 200 uri = /network-bootstore-config
19622024-10-17T15:12:24.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41816 req_id = 3dae9a47-9d92-4caf-9b96-6c449265c70c response_code = 200 uri = /network-bootstore-config
19632024-10-17T15:12:29.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33994
19642024-10-17T15:12:29.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33994 req_id = 6f7530dd-cc09-4a82-99ca-309612b12bd9 response_code = 200 uri = /v2p
19652024-10-17T15:12:29.091ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:838 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(4999943) }
19662024-10-17T15:12:29.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33994 req_id = c0faae78-f702-4ee8-9e58-a85b665d6a7e response_code = 204 uri = /v2p
19672024-10-17T15:12:29.211ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50919
19682024-10-17T15:12:29.223ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:555 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(4999943))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-4912745aa35", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e5e8ae12-301c-4bd4-907e-33852ba75a92, read_only_parent: Some(Volume { block_size: 512, id: 354ea118-8223-4352-ac40-e8e7f187eadb, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: b0e300d4-9c8e-48ce-87b5-eb33ef296479, key: Some("cl8ZRvj9/EoPgSjCps482o8HA8F51legaK6G1J3OZFQ="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e5e8ae12-301c-4bd4-907e-33852ba75a92, key: Some("rgXcssQwyKX6BgctsXQWmj9E19NJH0sRjwmhQoGifYA="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::13]:19002", "[fd00:1122:3344:101::14]:19000"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-4912745aa35" }] }), cloud_init_bytes: Some(..) } instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 metadata = InstanceMetadata { silo_id: 3ca9570d-92af-40f4-a3ec-b642e876d6bd, project_id: 5aa2038b-a998-40ab-a2c7-417f086345f2 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-17T15:12:29.211258900Z }
19692024-10-17T15:12:29.223ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:588 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 migration_id = None propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19702024-10-17T15:12:29.223ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1143 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 migration_id = None propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.1.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.1.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(4999943))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: c3c92cb2-43d9-4406-8970-79bed238a55b, kind: Instance { id: 278be560-06c0-46fe-9b25-af508f2fb075 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(4999943), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-4912745aa35", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: e5e8ae12-301c-4bd4-907e-33852ba75a92, read_only_parent: Some(Volume { block_size: 512, id: 354ea118-8223-4352-ac40-e8e7f187eadb, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: b0e300d4-9c8e-48ce-87b5-eb33ef296479, key: Some("cl8ZRvj9/EoPgSjCps482o8HA8F51legaK6G1J3OZFQ="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: e5e8ae12-301c-4bd4-907e-33852ba75a92, key: Some("rgXcssQwyKX6BgctsXQWmj9E19NJH0sRjwmhQoGifYA="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::17]:19001", "[fd00:1122:3344:101::13]:19002", "[fd00:1122:3344:101::14]:19000"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-4912745aa35" }] }), cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-17T15:12:29.211258900Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
19712024-10-17T15:12:29.226ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 15244 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50919 req_id = 23d67858-5f7d-4697-84d4-b65eae585fe6 response_code = 200 uri = /vmms/2a25a16d-1c98-4e6a-9f65-3fd935162963
19722024-10-17T15:12:29.364ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59551
19732024-10-17T15:12:29.365ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 843 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59551 req_id = ee444e5d-b1f2-4a3e-8a94-4280bec66c83 response_code = 200 uri = /vmms/2a25a16d-1c98-4e6a-9f65-3fd935162963/state
19742024-10-17T15:12:29.366ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 4999943 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
19752024-10-17T15:12:29.366ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
19762024-10-17T15:12:29.405ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963 file = illumos-utils/src/zone.rs:316 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19772024-10-17T15:12:29.428ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963 file = illumos-utils/src/zone.rs:354 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19782024-10-17T15:12:29.478ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62101
19792024-10-17T15:12:29.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62101 req_id = 1279e28e-5fb7-4a4f-bf3e-3c441848ff3c response_code = 200 uri = /vpc-routes
19802024-10-17T15:12:29.493ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {c3c92cb2-43d9-4406-8970-79bed238a55b: {10.151.1.175: {599f0db7-a669-41a9-b25b-54efc16d3edd}, 10.151.1.176: {599f0db7-a669-41a9-b25b-54efc16d3edd}}} }
19812024-10-17T15:12:30.648ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19822024-10-17T15:12:30.648ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963 zone file = illumos-utils/src/running_zone.rs:488 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19832024-10-17T15:12:30.825ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 29.246377ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19842024-10-17T15:12:30.864ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 88.48932ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19852024-10-17T15:12:30.967ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 263.694813ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19862024-10-17T15:12:31.246ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 500.91949ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19872024-10-17T15:12:31.759ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 845.824917ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19882024-10-17T15:12:32.437ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 23881 local_addr = [fd00:1122:3344:101::1]:44276 method = GET remote_addr = [fd00:1122:3344:101::d]:40319 req_id = b6e30c14-ae7d-4a32-8696-6fbafcb0eddf response_code = 200 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 uri = /a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
19892024-10-17T15:12:32.618ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 1.484600197s file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19902024-10-17T15:12:34.114ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 1.204138591s file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19912024-10-17T15:12:35.330ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 1.073077981s file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19922024-10-17T15:12:36.415ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 832.967758ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 zone = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
19932024-10-17T15:12:37.277ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963 file = sled-agent/src/instance.rs:1637 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19942024-10-17T15:12:37.287ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 58.067911ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19952024-10-17T15:12:37.355ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 149.06969ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19962024-10-17T15:12:37.515ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 175.60484ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
19972024-10-17T15:12:37.624ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36247
19982024-10-17T15:12:37.624ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100 local_addr = [fdb0:18c0:4d0d:9fb2::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:9fb2::2]:36247 req_id = 4d2b90ad-3015-4f5a-82b6-5f4a7b6b5a76 response_code = 200 uri = /baseboard
19992024-10-17T15:12:37.700ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 500.244749ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20002024-10-17T15:12:38.212ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963") failed: Property not found. retry in 996.568836ms file = illumos-utils/src/svc.rs:36 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20012024-10-17T15:12:39.221ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1646 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20022024-10-17T15:12:39.257ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1670 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20032024-10-17T15:12:39.395ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:835 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20042024-10-17T15:12:39.396ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1205 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
20052024-10-17T15:12:39.396ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9903313 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62101 req_id = f86a40f7-58f2-48b4-acee-b88709fb2a7a response_code = 204 uri = /eip-gateways
20062024-10-17T15:12:39.397ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-10-17T15:12:39.397844362Z } file = sled-agent/src/instance.rs:685 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20072024-10-17T15:12:39.397ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-17T15:12:39.397844362Z } propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20082024-10-17T15:12:39.397ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-17T15:12:39.397844362Z }, migration_in: None, migration_out: None }
20092024-10-17T15:12:39.466ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-10-17T15:12:39.466923150Z } file = sled-agent/src/instance.rs:685 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20102024-10-17T15:12:39.466ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-17T15:12:39.466923150Z } propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20112024-10-17T15:12:39.466ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-17T15:12:39.466923150Z }, migration_in: None, migration_out: None }
20122024-10-17T15:12:39.483ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 295 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62101 req_id = 5c7b6118-ba47-49cc-84c6-aee46dd2edb2 response_code = 204 uri = /vpc-routes
20132024-10-17T15:12:52.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59538
20142024-10-17T15:12:52.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 592 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59538 req_id = 55ff1945-9e37-405c-b709-37e9d59a0541 response_code = 200 uri = /vmms/2a25a16d-1c98-4e6a-9f65-3fd935162963/state
20152024-10-17T15:12:52.034ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-10-17T15:12:52.034096701Z } file = sled-agent/src/instance.rs:685 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20162024-10-17T15:12:52.034ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-17T15:12:52.034096701Z } propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20172024-10-17T15:12:52.034ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-17T15:12:52.034096701Z }, migration_in: None, migration_out: None }
20182024-10-17T15:12:52.105ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-10-17T15:12:52.105118345Z } file = sled-agent/src/instance.rs:685 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20192024-10-17T15:12:52.105ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:707 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-10-17T15:12:52.105118345Z } propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20202024-10-17T15:12:52.105ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:721 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20212024-10-17T15:12:52.105ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20222024-10-17T15:12:52.106ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 5263310137507237121 n_samples = 6 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
20232024-10-17T15:12:52.106ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 12212669941078320311 n_samples = 6 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
20242024-10-17T15:12:52.108ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20252024-10-17T15:12:52.109ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/2af104c1-ee2c-49fb-8805-3ed22aa08cdd/debug/bundle/zone", "/pool/int/34423522-8473-4e67-b21d-dccdbba2931c/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/86d90d1a-236b-48f7-9d13-e0e08c6f416e/crypt/debug", "/pool/ext/6bbd0bbe-4092-4d91-b621-d8484bd8935a/crypt/debug", "/pool/ext/13507129-93b4-4cdb-aa3e-5a74ab4d7515/crypt/debug", "/pool/ext/d1ed9d1c-3bd6-4d1d-b6b7-a90fc43bbfe0/crypt/debug", "/pool/ext/df85cc60-c9de-42c1-9f2e-5eeb9b5fc01f/crypt/debug", "/pool/ext/e5d462a9-5343-4c50-9dc4-2adbb2b66d5e/crypt/debug", "/pool/ext/0bf5df29-c2fa-4bc7-afc1-14e4a6edbeaf/crypt/debug", "/pool/ext/6dc43632-8304-473f-ba75-413bbd171b4c/crypt/debug", "/pool/ext/6931ef6e-c78c-4d61-ab93-92f35309d55f/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963
20262024-10-17T15:12:52.748ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963", bundle_id: 367078eb-5d8d-452a-a154-2d118a3953a3 }, time_created: 2024-10-17T15:12:52.109842914Z, version: 0, cause: TerminatedInstance }
20272024-10-17T15:12:52.748ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_2a25a16d-1c98-4e6a-9f65-3fd935162963 file = sled-agent/src/instance.rs:935 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20282024-10-17T15:12:53.997ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35035
20292024-10-17T15:12:53.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35035 req_id = 04e4cba5-1340-4a23-8e26-f11c2947934e response_code = 200 uri = /vpc-routes
20302024-10-17T15:12:54.008ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56110
20312024-10-17T15:12:54.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64133
20322024-10-17T15:12:54.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64133 req_id = 32cd5877-8bc1-4551-9469-50995c8a119f response_code = 200 uri = /v2p
20332024-10-17T15:12:54.101ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {c3c92cb2-43d9-4406-8970-79bed238a55b: {10.151.1.175: {599f0db7-a669-41a9-b25b-54efc16d3edd}, 10.151.1.176: {599f0db7-a669-41a9-b25b-54efc16d3edd}}} }
20342024-10-17T15:12:54.101ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35035 req_id = ab6d8a13-aa84-4d92-a9be-bd7e429e7be0 response_code = 204 uri = /eip-gateways
20352024-10-17T15:12:54.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 328 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35035 req_id = 049b8d83-05e1-417f-bcde-46c78ebd89a4 response_code = 204 uri = /vpc-routes
20362024-10-17T15:12:54.290ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38144
20372024-10-17T15:12:54.290ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20382024-10-17T15:12:54.336ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20392024-10-17T15:12:54.405ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114353 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38144 req_id = e8aeb75c-460b-493c-a65d-f1b14847ed83 response_code = 204 uri = /switch-ports
20402024-10-17T15:12:54.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42579
20412024-10-17T15:12:54.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47670
20422024-10-17T15:12:54.405ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57621
20432024-10-17T15:12:54.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4960 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38144 req_id = a85e3e60-1b7c-4870-978a-1d8318fb4550 response_code = 200 uri = /network-bootstore-config
20442024-10-17T15:12:54.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 211 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57621 req_id = 724875a0-61e3-4d99-89a9-5d6f8d9301c7 response_code = 200 uri = /vpc-routes
20452024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
20462024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20472024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20482024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20492024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20502024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
20512024-10-17T15:12:54.411ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20522024-10-17T15:12:54.411ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 770 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47670 req_id = 0e43b131-4d1b-4438-a0bc-3e50cd6c6503 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
20532024-10-17T15:12:54.462ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {c3c92cb2-43d9-4406-8970-79bed238a55b: {10.151.1.175: {599f0db7-a669-41a9-b25b-54efc16d3edd}, 10.151.1.176: {599f0db7-a669-41a9-b25b-54efc16d3edd}}} }
20542024-10-17T15:12:54.462ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2672 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57621 req_id = b1465ec3-29d5-4abb-a4a4-b0aa49ad282c response_code = 204 uri = /eip-gateways
20552024-10-17T15:12:54.530ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45489
20562024-10-17T15:12:54.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1606 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45489 req_id = 4ecdf83b-908a-4d46-8836-c3ac8071e67c response_code = 200 uri = /v2p
20572024-10-17T15:12:54.540ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38462
20582024-10-17T15:12:54.541ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 531 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38462 req_id = 1b35d609-0dd8-4241-aa7a-da079c6ce0e9 response_code = 200 uri = /vpc-routes
20592024-10-17T15:12:54.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2456 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57621 req_id = 78850c1d-57cb-4589-b587-04fbf4a85a3d response_code = 204 uri = /vpc-routes
20602024-10-17T15:12:54.568ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38558
20612024-10-17T15:12:54.595ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63550
20622024-10-17T15:12:54.596ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20632024-10-17T15:12:54.645ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20642024-10-17T15:12:54.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119880 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63550 req_id = d7167267-81e6-49ce-aa02-6cd20fcef6f6 response_code = 204 uri = /switch-ports
20652024-10-17T15:12:54.715ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {c3c92cb2-43d9-4406-8970-79bed238a55b: {10.151.1.176: {599f0db7-a669-41a9-b25b-54efc16d3edd}, 10.151.1.175: {599f0db7-a669-41a9-b25b-54efc16d3edd}}} }
20662024-10-17T15:12:54.715ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38462 req_id = 91c78f1f-9185-443f-8033-5519a69f34b8 response_code = 204 uri = /eip-gateways
20672024-10-17T15:12:54.716ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 232 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63550 req_id = da4ba0e5-5e60-429b-acc2-ba015777b0fd response_code = 200 uri = /network-bootstore-config
20682024-10-17T15:12:54.763ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42690
20692024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
20702024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20712024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20722024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20732024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
20742024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
20752024-10-17T15:12:54.763ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
20762024-10-17T15:12:54.763ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 473 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42690 req_id = f1751f67-26f0-40be-bfb4-3c0c2a565688 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
20772024-10-17T15:12:54.825ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20782024-10-17T15:12:54.846ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(4999943), kind: System }
20792024-10-17T15:12:54.846ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(4999943), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
20802024-10-17T15:12:54.847ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-17T15:12:54.846982839Z }, migration_in: None, migration_out: None }
20812024-10-17T15:12:54.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41920
20822024-10-17T15:12:54.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41920 req_id = 3f72d2c6-e936-4af9-9ccf-5f3ef9fb4ba0 response_code = 200 uri = /v2p
20832024-10-17T15:12:54.884ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:838 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(4999943) }
20842024-10-17T15:12:54.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41920 req_id = ca40ab2e-2d5f-43bf-b8c1-1565bb2ec432 response_code = 204 uri = /v2p
20852024-10-17T15:12:54.900ZWARNSledAgent (PortManager): ignoring route ResolvedVpcRouteSet { id: RouterId { vni: Vni( 4999943, ), kind: Custom( V4( Ipv4Net { addr: 172.30.0.0, width: 22, }, ), ), }, version: None, routes: {}, } file = illumos-utils/src/opte/port_manager.rs:456
20862024-10-17T15:12:54.900ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 249 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38462 req_id = 843cfe21-1aa6-4fdd-a26c-68b48f0119d7 response_code = 204 uri = /vpc-routes
20872024-10-17T15:12:54.913ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:605 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-17T15:12:54.846982839Z }, migration_in: None, migration_out: None }
20882024-10-17T15:12:54.917ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:872 instance_id = 278be560-06c0-46fe-9b25-af508f2fb075 propolis_id = 2a25a16d-1c98-4e6a-9f65-3fd935162963
20892024-10-17T15:12:54.926ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46256
20902024-10-17T15:12:54.926ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20912024-10-17T15:12:54.968ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20922024-10-17T15:12:55.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 107052 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46256 req_id = 5d3b0c9a-7475-4b3d-848b-b9dc12820050 response_code = 204 uri = /switch-ports
20932024-10-17T15:12:55.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 286 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46256 req_id = 91234be5-264c-4022-9235-086da63a95b9 response_code = 200 uri = /network-bootstore-config
20942024-10-17T15:12:55.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1025847 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56110 req_id = 1e765329-44d3-4706-b33b-8b061b221869 response_code = 200 uri = /vmms/2a25a16d-1c98-4e6a-9f65-3fd935162963/state
20952024-10-17T15:12:55.034ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Not Found error_message_internal = VMM with ID 2a25a16d-1c98-4e6a-9f65-3fd935162963 not found file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 465654 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38558 req_id = a9af92c3-c720-4602-8831-369bc927f742 response_code = 404 uri = /vmms/2a25a16d-1c98-4e6a-9f65-3fd935162963/state
20962024-10-17T15:12:55.034ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Not Found error_message_internal = VMM with ID 2a25a16d-1c98-4e6a-9f65-3fd935162963 not found file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 624100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42579 req_id = 5fb01687-3e2f-488f-82fb-52428c2e82ca response_code = 404 uri = /vmms/2a25a16d-1c98-4e6a-9f65-3fd935162963/state
20972024-10-17T15:12:55.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42959
20982024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6d772bd8-afca-47c9-801c-49b18dffb0e7, kind: Service { id: d8aecb7f-3c94-4690-8de6-10380087d1f0 }, name: Name("external-dns-d8aecb7f-3c94-4690-8de6-10380087d1f0"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 178, 199])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6effcc2-2cd2-485f-8ff3-7925310f2295, kind: Service { id: 869dbbb9-e780-4452-80fd-27d18cfc03d9 }, name: Name("external-dns-869dbbb9-e780-4452-80fd-27d18cfc03d9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 197, 237])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46f0962a-0b66-4dce-b032-30a146c8b1a8, kind: Service { id: 9b355ea2-74f1-408f-acfc-0884fd332b64 }, name: Name("nexus-9b355ea2-74f1-408f-acfc-0884fd332b64"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 160, 83])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9b09b11f-c188-4c11-8ce3-d650ad9f0dd9, kind: Service { id: 282d1bc9-0cd4-4794-93fb-67f38865892f }, name: Name("nexus-282d1bc9-0cd4-4794-93fb-67f38865892f"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 242])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a0263431-08f3-4038-a052-4b9f35593548, kind: Service { id: 9125336f-8bba-4da8-a415-ccae7d60d01b }, name: Name("nexus-9125336f-8bba-4da8-a415-ccae7d60d01b"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 177, 199])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
20992024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21002024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21012024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21022024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
21032024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
21042024-10-17T15:12:55.084ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
21052024-10-17T15:12:55.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 354 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:42959 req_id = 0b5448f6-5f64-4b18-9fc2-185d6ce0ce9a response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
21062024-10-17T15:12:55.463ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41324
21072024-10-17T15:12:55.463ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41324 req_id = ab62abf3-d9ed-44f1-b56f-05f78135a3da response_code = 200 uri = /vpc-routes
21082024-10-17T15:12:55.478ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {c3c92cb2-43d9-4406-8970-79bed238a55b: {10.151.1.176: {599f0db7-a669-41a9-b25b-54efc16d3edd}, 10.151.1.175: {599f0db7-a669-41a9-b25b-54efc16d3edd}}} }
21092024-10-17T15:12:55.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41324 req_id = 32d7cd13-0577-4752-bfe0-518cb2c35a2a response_code = 204 uri = /eip-gateways
21102024-10-17T15:12:55.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41324 req_id = 04743971-d8e0-4071-8def-185176d92d3e response_code = 204 uri = /vpc-routes
21112024-10-17T15:12:55.620ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60290
21122024-10-17T15:12:55.620ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 130 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60290 req_id = 0b7b4fcb-3c2b-468c-a9de-23f5279f86b8 response_code = 200 uri = /v2p
21132024-10-17T15:12:55.621ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:878 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(4999943) }
21142024-10-17T15:12:55.621ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:60290 req_id = 439d51e1-e1cb-4c1c-8c43-db185869ce9b response_code = 204 uri = /v2p
21152024-10-17T15:12:56.884ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46446
21162024-10-17T15:12:56.884ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46446 req_id = 60d3277e-e378-4e83-9046-a4e34747d1be response_code = 200 uri = /v2p
21172024-10-17T15:12:57.032ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59886
21182024-10-17T15:12:57.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 64 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59886 req_id = a1c2e593-1e2e-4e89-ae2a-21e85556c051 response_code = 200 uri = /vpc-routes
21192024-10-17T15:12:57.046ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
21202024-10-17T15:12:57.046ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1205 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4
21212024-10-17T15:12:57.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 309 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59886 req_id = a18247a0-20fa-40c0-9416-3659153d1434 response_code = 204 uri = /eip-gateways
21222024-10-17T15:12:57.054ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 341 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59886 req_id = 28c8f3aa-9324-4a70-bc07-735c84e5537b response_code = 204 uri = /vpc-routes
21232024-10-17T15:12:57.844ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50710
21242024-10-17T15:12:57.844ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50710 req_id = f7558a35-c08a-447d-91bd-a36c39a3dabe response_code = 200 uri = /vpc-routes
21252024-10-17T15:12:57.856ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1179 sled_id = a66de6c2-c7a5-4a6e-aa75-a912ee2f8cf4 values = ExternalIpGatewayMap { mappings: {} }
21262024-10-17T15:12:57.856ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50710 req_id = e7058d44-599d-425a-b68c-ce4f842df2d8 response_code = 204 uri = /eip-gateways
21272024-10-17T15:12:57.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:50710 req_id = 815bacd8-6ecb-4ce4-bce5-829f9dc221fe response_code = 204 uri = /vpc-routes