Max level shown:
1 [ Oct 30 17:47:15 Enabled. ]
2 [ Oct 30 17:47:15 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 30 17:47:16 Method "start" exited with status 0. ]
4 [ Oct 30 17:47:16 Rereading configuration. ]
5 [ Oct 30 17:47:16 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-30T17:47:16.015ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-30T17:47:16.015ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-30T17:47:16.067ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-30T17:47:16.067ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-30T17:47:16.180ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-30T17:47:16.189ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-30T17:47:16.198ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-30T17:47:16.797ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-10-30T17:47:16.933ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-30T17:47:16.933ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-30T17:47:16.933ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-30T17:47:16.933ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-10-30T17:47:16.933ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-10-30T17:47:16.933ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-30T17:47:16.936ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-30T17:47:16.936ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-30T17:47:16.936ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-30T17:47:16.936ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-30T17:47:16.936ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-30T17:47:16.936ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-30T17:47:16.936ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-30T17:47:16.936ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-30T17:47:16.936ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-30T17:47:16.947ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-30T17:47:16.947ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool)
322024-10-30T17:47:17.072ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal }
332024-10-30T17:47:17.577ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-30T17:47:17.577ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-30T17:47:17.577ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-30T17:47:17.577ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-30T17:47:17.577ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
382024-10-30T17:47:17.578ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
392024-10-30T17:47:17.578ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
402024-10-30T17:47:17.578ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
412024-10-30T17:47:17.578ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
422024-10-30T17:47:17.578ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
432024-10-30T17:47:17.587ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
442024-10-30T17:47:17.587ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool)
452024-10-30T17:47:17.698ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal }
462024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
472024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
482024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
492024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
502024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
512024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
522024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
532024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
542024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
552024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
562024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
572024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
582024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
592024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
612024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
622024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
632024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
642024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
652024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
682024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
692024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
702024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
712024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
722024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
732024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
742024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
752024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
762024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
772024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
782024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
792024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
802024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
812024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
822024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
832024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
842024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
852024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
862024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
872024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
882024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
892024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
902024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
922024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
932024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
942024-10-30T17:47:18.152ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
952024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
962024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
972024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
982024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
992024-10-30T17:47:18.152ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
1002024-10-30T17:47:18.152ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
1012024-10-30T17:47:18.152ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
1022024-10-30T17:47:18.152ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1032024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1042024-10-30T17:47:18.152ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1052024-10-30T17:47:18.152ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1062024-10-30T17:47:18.152ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1072024-10-30T17:47:18.153ZINFOSledAgent (bootstore): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-10-30T17:47:18.153ZINFOSledAgent (bootstore): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-10-30T17:47:18.153ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1102024-10-30T17:47:18.153ZINFOSledAgent (bootstore): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1112024-10-30T17:47:18.153ZINFOSledAgent (bootstore): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1122024-10-30T17:47:18.153ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1132024-10-30T17:47:18.153ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1142024-10-30T17:47:18.153ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1152024-10-30T17:47:18.162ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:733
1162024-10-30T17:47:18.162ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1297, tv_nsec: 185791511 } time_to_next_cleanup = 599.999937518s
1172024-10-30T17:47:18.162ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3908
1182024-10-30T17:47:18.163ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1192024-10-30T17:47:18.165ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/sled-agent-request.json file = common/src/ledger.rs:188
1202024-10-30T17:47:18.165ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/sled-agent-request.json file = common/src/ledger.rs:188
1212024-10-30T17:47:18.166ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1222024-10-30T17:47:18.167ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1232024-10-30T17:47:18.168ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1073
1242024-10-30T17:47:18.168ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1252024-10-30T17:47:18.168ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1262024-10-30T17:47:18.168ZINFOSledAgent (RSS): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1272024-10-30T17:47:18.168ZINFOSledAgent (RSS): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1282024-10-30T17:47:18.168ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1136
1292024-10-30T17:47:18.168ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1302024-10-30T17:47:18.168ZINFOSledAgent (RSS): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/rss-sled-plan.json file = common/src/ledger.rs:188
1312024-10-30T17:47:18.168ZINFOSledAgent (RSS): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/rss-sled-plan.json file = common/src/ledger.rs:188
1322024-10-30T17:47:18.168ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1178
1332024-10-30T17:47:18.168ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1342024-10-30T17:47:18.168ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1352024-10-30T17:47:18.168ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1362024-10-30T17:47:18.168ZINFOSledAgent (RSS): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1372024-10-30T17:47:18.168ZINFOSledAgent (RSS): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1382024-10-30T17:47:18.169ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 5f1b3995-6708-40e4-9d28-234b43b9cddc, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 297da6c0-3dc5-4f20-9a57-48a339550273 (sled), rack_id: 5f1b3995-6708-40e4-9d28-234b43b9cddc, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1392024-10-30T17:47:18.169ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1212
1402024-10-30T17:47:18.169ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1412024-10-30T17:47:18.169ZINFOSledAgent (bootstore): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1422024-10-30T17:47:18.169ZINFOSledAgent (bootstore): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1432024-10-30T17:47:18.169ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1442024-10-30T17:47:18.169ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 297da6c0-3dc5-4f20-9a57-48a339550273 (sled), rack_id: 5f1b3995-6708-40e4-9d28-234b43b9cddc, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1452024-10-30T17:47:18.169ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4205
1462024-10-30T17:47:18.169ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4587
1472024-10-30T17:47:18.174ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1482024-10-30T17:47:18.180ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1492024-10-30T17:47:18.180ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1502024-10-30T17:47:18.185ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1512024-10-30T17:47:18.185ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1522024-10-30T17:47:18.185ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:a02a::1]:43145
1532024-10-30T17:47:18.185ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 297da6c0-3dc5-4f20-9a57-48a339550273 (sled), rack_id: 5f1b3995-6708-40e4-9d28-234b43b9cddc, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1542024-10-30T17:47:18.185ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1552024-10-30T17:47:18.185ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1562024-10-30T17:47:18.185ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
1572024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1582024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1592024-10-30T17:47:18.185ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1602024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1612024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1622024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1632024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1642024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1652024-10-30T17:47:18.185ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1662024-10-30T17:47:18.185ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1672024-10-30T17:47:18.201ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1682024-10-30T17:47:18.222ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1692024-10-30T17:47:18.239ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1702024-10-30T17:47:18.246ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:401 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
1712024-10-30T17:47:18.246ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1722024-10-30T17:47:18.246ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:413 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
1732024-10-30T17:47:18.246ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1742024-10-30T17:47:18.249ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 93.670893ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "47691df3-2f72-4670-9788-58349ef3b52f", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "47691df3-2f72-4670-9788-58349ef3b52f" } file = clients/ddm-admin-client/src/lib.rs:177
1752024-10-30T17:47:18.258ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_3485ccb7-7923-4269-8b7b-16caa41fa8d8/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1762024-10-30T17:47:18.286ZINFOSledAgent: successfully created encrypted zvol: name="oxi_3485ccb7-7923-4269-8b7b-16caa41fa8d8/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1772024-10-30T17:47:18.286ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_3485ccb7-7923-4269-8b7b-16caa41fa8d8/swap" file = sled-agent/src/swap_device.rs:96
1782024-10-30T17:47:18.291ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:428 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
1792024-10-30T17:47:18.291ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1802024-10-30T17:47:18.291ZINFOSledAgent (BackingFs): Ensuring dataset oxi_3485ccb7-7923-4269-8b7b-16caa41fa8d8/backing/fmd file = sled-agent/src/backing_fs.rs:138
1812024-10-30T17:47:18.345ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-30T17:47:18.346ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 135.796157ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "446e3056-0981-4c33-97ce-e77bd6c47a57", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "446e3056-0981-4c33-97ce-e77bd6c47a57" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-30T17:47:18.388ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1842024-10-30T17:47:18.482ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1852024-10-30T17:47:18.483ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 170.861589ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "059310f6-06de-49f3-a2e9-f1822a2166cf", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "059310f6-06de-49f3-a2e9-f1822a2166cf" } file = clients/ddm-admin-client/src/lib.rs:177
1862024-10-30T17:47:18.608ZINFOSledAgent (BackingFs): Mounting oxi_3485ccb7-7923-4269-8b7b-16caa41fa8d8/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1872024-10-30T17:47:18.621ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1882024-10-30T17:47:18.621ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1892024-10-30T17:47:18.621ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1902024-10-30T17:47:18.621ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1912024-10-30T17:47:18.655ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1922024-10-30T17:47:18.655ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 199.993817ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "084551be-ff82-4c41-9bde-0a8ece27021d", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "084551be-ff82-4c41-9bde-0a8ece27021d" } file = clients/ddm-admin-client/src/lib.rs:177
1932024-10-30T17:47:18.856ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1942024-10-30T17:47:18.857ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 224.310916ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e65a64df-45b7-4227-8504-be17e1df2d61", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e65a64df-45b7-4227-8504-be17e1df2d61" } file = clients/ddm-admin-client/src/lib.rs:177
1952024-10-30T17:47:19.082ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1962024-10-30T17:47:19.083ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 363.10272ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "44942025-d303-4db1-8943-62f0aee79c8a", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:18 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "44942025-d303-4db1-8943-62f0aee79c8a" } file = clients/ddm-admin-client/src/lib.rs:177
1972024-10-30T17:47:19.364ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
1982024-10-30T17:47:19.369ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:56704 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
1992024-10-30T17:47:19.369ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2002024-10-30T17:47:19.369ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:56704 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 297da6c0-3dc5-4f20-9a57-48a339550273 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2012024-10-30T17:47:19.370ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 258.404934ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2022024-10-30T17:47:19.371ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2032024-10-30T17:47:19.371ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2042024-10-30T17:47:19.371ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:917 underlay_address = fd00:1122:3344:101::1
2052024-10-30T17:47:19.371ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/resolver/src/resolver.rs:111
2062024-10-30T17:47:19.447ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2072024-10-30T17:47:19.630ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 722.341332ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2082024-10-30T17:47:20.354ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.322508213s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2092024-10-30T17:47:21.678ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.673926081s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2102024-10-30T17:47:21.867ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2663
2112024-10-30T17:47:21.867ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2881
2122024-10-30T17:47:21.867ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2533
2132024-10-30T17:47:21.867ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2580
2142024-10-30T17:47:21.867ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2962
2152024-10-30T17:47:21.867ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3004
2162024-10-30T17:47:21.869ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2806
2172024-10-30T17:47:21.870ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2574
2182024-10-30T17:47:21.870ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='5f95daab-fbcb-4007-b6a6-2d0ad673f309'/> <propval type="astring" name="rack_id" value='5f1b3995-6708-40e4-9d28-234b43b9cddc'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe3e:8b2b'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='297da6c0-3dc5-4f20-9a57-48a339550273'/> <propval type="astring" name="rack_id" value='5f1b3995-6708-40e4-9d28-234b43b9cddc'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='297da6c0-3dc5-4f20-9a57-48a339550273'/> <propval type="astring" name="rack_uuid" value='5f1b3995-6708-40e4-9d28-234b43b9cddc'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='297da6c0-3dc5-4f20-9a57-48a339550273'/> <propval type="astring" name="rack_uuid" value='5f1b3995-6708-40e4-9d28-234b43b9cddc'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2192024-10-30T17:47:21.870ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2202024-10-30T17:47:22.124ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 66.930266ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-30T17:47:22.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 55.700932ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-10-30T17:47:22.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 157.283823ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-30T17:47:22.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 411.145277ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2242024-10-30T17:47:23.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 737.181995ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-10-30T17:47:23.353ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.329773196s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2262024-10-30T17:47:24.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.158234045s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-30T17:47:25.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 939.027547ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2282024-10-30T17:47:25.685ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 6.477869076s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2292024-10-30T17:47:26.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.152517889s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2302024-10-30T17:47:28.056ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2312024-10-30T17:47:28.056ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2322024-10-30T17:47:28.056ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2332024-10-30T17:47:28.056ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2342024-10-30T17:47:28.056ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2352024-10-30T17:47:28.056ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2362024-10-30T17:47:28.056ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:a02a::1]:43145
2372024-10-30T17:47:28.056ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, rib_priority: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2382024-10-30T17:47:28.056ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3908
2392024-10-30T17:47:28.057ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2402024-10-30T17:47:28.057ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2412024-10-30T17:47:28.283ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2422024-10-30T17:47:28.283ZINFOSledAgent (RSS): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2432024-10-30T17:47:28.283ZINFOSledAgent (RSS): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2442024-10-30T17:47:28.283ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-30T17:47:28.283ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2462024-10-30T17:47:28.283ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2472024-10-30T17:47:28.289ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4229 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2482024-10-30T17:47:28.289ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4245
2492024-10-30T17:47:28.289ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2502024-10-30T17:47:28.578ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35157
2512024-10-30T17:47:28.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2522024-10-30T17:47:28.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2532024-10-30T17:47:28.579ZINFOSledAgent (ServiceManager): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:188
2542024-10-30T17:47:28.579ZINFOSledAgent (ServiceManager): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:188
2552024-10-30T17:47:28.579ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 313 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:35157 req_id = c907deea-3553-401b-8c1e-e57efd073f5a response_code = 200 uri = /inventory
2562024-10-30T17:47:28.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38409
2572024-10-30T17:47:28.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:38409 req_id = 2b37a5a7-455e-4096-b91e-5c45be718536 response_code = 200 uri = /sled-role
2582024-10-30T17:47:28.611ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2592024-10-30T17:47:28.611ZINFOSledAgent (RSS): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2602024-10-30T17:47:28.611ZINFOSledAgent (RSS): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2612024-10-30T17:47:28.612ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:849
2622024-10-30T17:47:28.643ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 7e0d1d8f-0613-4639-b431-9afe58499e31, pool_id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 7b7f716b-e6fd-44b3-8dd0-21c850a4c57a, pool_id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: a8128710-3ba6-4a8a-a6ac-961172721267, pool_id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: f30ec10b-43b4-4722-ad14-0c12ee43f33d, pool_id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 28aca8aa-6952-4bce-912a-012e1a5f7306, pool_id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: b566a485-4cee-4068-9d26-7a165edc415b, pool_id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 3fbbac73-7b09-4183-b965-839f95fdaa7f, pool_id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 05e9b1d8-a137-49cb-af4f-2a2d9bb9abd9, pool_id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 2e5d6994-1956-4ef0-913f-5b28c524aba0, pool_id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2632024-10-30T17:47:28.644ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52663
2642024-10-30T17:47:28.644ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:874 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2652024-10-30T17:47:28.644ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 7e0d1d8f-0613-4639-b431-9afe58499e31, pool_id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 7b7f716b-e6fd-44b3-8dd0-21c850a4c57a, pool_id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: a8128710-3ba6-4a8a-a6ac-961172721267, pool_id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: f30ec10b-43b4-4722-ad14-0c12ee43f33d, pool_id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: 28aca8aa-6952-4bce-912a-012e1a5f7306, pool_id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: b566a485-4cee-4068-9d26-7a165edc415b, pool_id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 3fbbac73-7b09-4183-b965-839f95fdaa7f, pool_id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 05e9b1d8-a137-49cb-af4f-2a2d9bb9abd9, pool_id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 2e5d6994-1956-4ef0-913f-5b28c524aba0, pool_id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2662024-10-30T17:47:28.644ZINFOSledAgent (StorageManager): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2672024-10-30T17:47:28.644ZINFOSledAgent (StorageManager): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2682024-10-30T17:47:28.644ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2692024-10-30T17:47:28.644ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2702024-10-30T17:47:28.644ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2712024-10-30T17:47:28.644ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2722024-10-30T17:47:28.644ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2732024-10-30T17:47:28.784ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2742024-10-30T17:47:28.784ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool)
2752024-10-30T17:47:29.247ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/probe_manager.rs:230 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2762024-10-30T17:47:29.467ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2772024-10-30T17:47:29.469ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5561917105458027348 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2782024-10-30T17:47:29.469ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 13283554707908944394 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
2792024-10-30T17:47:29.522ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }
2802024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2812024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2822024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2832024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2842024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2852024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2862024-10-30T17:47:29.530ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2872024-10-30T17:47:29.571ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2882024-10-30T17:47:29.578ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5c811413-1d4e-403e-b8bb-331b2268c476/crypt/zone file = sled-storage/src/dataset.rs:280
2892024-10-30T17:47:29.786ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5c811413-1d4e-403e-b8bb-331b2268c476 file = sled-storage/src/dataset.rs:366
2902024-10-30T17:47:29.793ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2912024-10-30T17:47:29.794ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2922024-10-30T17:47:29.794ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2932024-10-30T17:47:29.794ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2942024-10-30T17:47:29.794ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2952024-10-30T17:47:29.803ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2962024-10-30T17:47:29.803ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool)
2972024-10-30T17:47:29.918ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }
2982024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2992024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
3002024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
3012024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
3022024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3032024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3042024-10-30T17:47:29.925ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3052024-10-30T17:47:29.966ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3062024-10-30T17:47:29.974ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_7c337fba-b737-4f90-a327-065b4f3b4476/crypt/zone file = sled-storage/src/dataset.rs:280
3072024-10-30T17:47:30.176ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_7c337fba-b737-4f90-a327-065b4f3b4476 file = sled-storage/src/dataset.rs:366
3082024-10-30T17:47:30.183ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3092024-10-30T17:47:30.183ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3102024-10-30T17:47:30.183ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3112024-10-30T17:47:30.183ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3122024-10-30T17:47:30.183ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3132024-10-30T17:47:30.192ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3142024-10-30T17:47:30.192ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool)
3152024-10-30T17:47:30.307ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }
3162024-10-30T17:47:30.314ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3172024-10-30T17:47:30.314ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3182024-10-30T17:47:30.314ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3192024-10-30T17:47:30.315ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3202024-10-30T17:47:30.315ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3212024-10-30T17:47:30.315ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3222024-10-30T17:47:30.315ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3232024-10-30T17:47:30.319ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/probe_manager.rs:230 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
3242024-10-30T17:47:30.357ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3252024-10-30T17:47:30.365ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_46e37abb-39ab-4652-aa9d-61a2079024f1/crypt/zone file = sled-storage/src/dataset.rs:280
3262024-10-30T17:47:30.568ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_46e37abb-39ab-4652-aa9d-61a2079024f1 file = sled-storage/src/dataset.rs:366
3272024-10-30T17:47:30.575ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3282024-10-30T17:47:30.575ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3292024-10-30T17:47:30.575ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3302024-10-30T17:47:30.575ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3312024-10-30T17:47:30.575ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3322024-10-30T17:47:30.585ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3332024-10-30T17:47:30.585ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool)
3342024-10-30T17:47:30.701ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }
3352024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3362024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3372024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3382024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3392024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3402024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3412024-10-30T17:47:30.709ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3422024-10-30T17:47:30.750ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3432024-10-30T17:47:30.758ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8b4477e5-c445-4846-a713-78ddac5696a7/crypt/zone file = sled-storage/src/dataset.rs:280
3442024-10-30T17:47:30.959ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8b4477e5-c445-4846-a713-78ddac5696a7 file = sled-storage/src/dataset.rs:366
3452024-10-30T17:47:30.966ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3462024-10-30T17:47:30.966ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3472024-10-30T17:47:30.966ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3482024-10-30T17:47:30.966ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3492024-10-30T17:47:30.966ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3502024-10-30T17:47:30.976ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3512024-10-30T17:47:30.976ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool)
3522024-10-30T17:47:31.085ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }
3532024-10-30T17:47:31.092ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3542024-10-30T17:47:31.093ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3552024-10-30T17:47:31.093ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3562024-10-30T17:47:31.093ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3572024-10-30T17:47:31.093ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3582024-10-30T17:47:31.093ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3592024-10-30T17:47:31.093ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3602024-10-30T17:47:31.133ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3612024-10-30T17:47:31.140ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_09d27db0-c064-443e-8e36-e29bb4d8f58d/crypt/zone file = sled-storage/src/dataset.rs:280
3622024-10-30T17:47:31.320ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/probe_manager.rs:230 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
3632024-10-30T17:47:31.343ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_09d27db0-c064-443e-8e36-e29bb4d8f58d file = sled-storage/src/dataset.rs:366
3642024-10-30T17:47:31.351ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3652024-10-30T17:47:31.351ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3662024-10-30T17:47:31.351ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3672024-10-30T17:47:31.351ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3682024-10-30T17:47:31.351ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3692024-10-30T17:47:31.360ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3702024-10-30T17:47:31.360ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool)
3712024-10-30T17:47:31.475ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External }
3722024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3732024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3742024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3752024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3762024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3772024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3782024-10-30T17:47:31.482ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3792024-10-30T17:47:31.523ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3802024-10-30T17:47:31.531ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9fecadf7-1f9f-4c05-baeb-3ca812eb625c/crypt/zone file = sled-storage/src/dataset.rs:280
3812024-10-30T17:47:31.737ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9fecadf7-1f9f-4c05-baeb-3ca812eb625c file = sled-storage/src/dataset.rs:366
3822024-10-30T17:47:31.744ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3832024-10-30T17:47:31.744ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3842024-10-30T17:47:31.744ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3852024-10-30T17:47:31.744ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3862024-10-30T17:47:31.744ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3872024-10-30T17:47:31.754ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3882024-10-30T17:47:31.754ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = c84ed810-b8fd-4283-8511-603d43b629a7 (zpool)
3892024-10-30T17:47:31.864ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External }
3902024-10-30T17:47:31.871ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3912024-10-30T17:47:31.871ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3922024-10-30T17:47:31.871ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3932024-10-30T17:47:31.871ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3942024-10-30T17:47:31.871ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3952024-10-30T17:47:31.871ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3962024-10-30T17:47:31.872ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3972024-10-30T17:47:31.913ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3982024-10-30T17:47:31.920ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c84ed810-b8fd-4283-8511-603d43b629a7/crypt/zone file = sled-storage/src/dataset.rs:280
3992024-10-30T17:47:32.142ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c84ed810-b8fd-4283-8511-603d43b629a7 file = sled-storage/src/dataset.rs:366
4002024-10-30T17:47:32.150ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
4012024-10-30T17:47:32.150ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4022024-10-30T17:47:32.150ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4032024-10-30T17:47:32.150ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4042024-10-30T17:47:32.150ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4052024-10-30T17:47:32.160ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4062024-10-30T17:47:32.160ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = f6c51041-d1f1-4026-854a-17c6531b176e (zpool)
4072024-10-30T17:47:32.280ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }
4082024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4092024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4102024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4112024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4122024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4132024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4142024-10-30T17:47:32.287ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4152024-10-30T17:47:32.291ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 11.024619642s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4162024-10-30T17:47:32.323ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/probe_manager.rs:230 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4172024-10-30T17:47:32.329ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4182024-10-30T17:47:32.337ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f6c51041-d1f1-4026-854a-17c6531b176e/crypt/zone file = sled-storage/src/dataset.rs:280
4192024-10-30T17:47:32.543ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f6c51041-d1f1-4026-854a-17c6531b176e file = sled-storage/src/dataset.rs:366
4202024-10-30T17:47:32.551ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4212024-10-30T17:47:32.551ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4222024-10-30T17:47:32.551ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4232024-10-30T17:47:32.551ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4242024-10-30T17:47:32.551ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4252024-10-30T17:47:32.561ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4262024-10-30T17:47:32.561ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool)
4272024-10-30T17:47:32.679ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External }
4282024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4292024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4302024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4312024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4322024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4332024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4342024-10-30T17:47:32.687ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4352024-10-30T17:47:32.728ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4362024-10-30T17:47:32.735ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_b47c81a2-215a-4a96-ada9-ce831b19ee43/crypt/zone file = sled-storage/src/dataset.rs:280
4372024-10-30T17:47:32.939ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_b47c81a2-215a-4a96-ada9-ce831b19ee43 file = sled-storage/src/dataset.rs:366
4382024-10-30T17:47:32.947ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4392024-10-30T17:47:32.947ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4402024-10-30T17:47:32.947ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4412024-10-30T17:47:32.947ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 3485ccb7-7923-4269-8b7b-16caa41fa8d8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: adb893fc-56e4-4ac7-a502-4c948ee9f9bd (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4422024-10-30T17:47:32.947ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4432024-10-30T17:47:32.947ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4442024-10-30T17:47:32.947ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4452024-10-30T17:47:32.947ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:878 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4462024-10-30T17:47:32.947ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4472024-10-30T17:47:32.947ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:904 generation = Generation(1) sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4482024-10-30T17:47:32.948ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:909 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4492024-10-30T17:47:32.948ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:914 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4502024-10-30T17:47:32.948ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4512024-10-30T17:47:32.948ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:920 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4522024-10-30T17:47:32.948ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:799 gen = Generation(1)
4532024-10-30T17:47:32.948ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:925 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4542024-10-30T17:47:32.948ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4304018 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:52663 req_id = 45fc352d-913e-4c1d-9d4c-0051670c25e3 response_code = 200 uri = /omicron-physical-disks
4552024-10-30T17:47:32.979ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4562024-10-30T17:47:32.979ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42875
4572024-10-30T17:47:32.979ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4582024-10-30T17:47:32.979ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4592024-10-30T17:47:33.040ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4602024-10-30T17:47:33.040ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4612024-10-30T17:47:33.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4622024-10-30T17:47:33.098ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4632024-10-30T17:47:33.156ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4642024-10-30T17:47:33.156ZINFOSledAgent (ServiceManager): No ledger in /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:188
4652024-10-30T17:47:33.156ZINFOSledAgent (ServiceManager): No ledger in /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:188
4662024-10-30T17:47:33.156ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4672024-10-30T17:47:33.199ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4682024-10-30T17:47:33.199ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4692024-10-30T17:47:33.199ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4702024-10-30T17:47:33.252ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58 file = illumos-utils/src/zone.rs:316
4712024-10-30T17:47:33.256ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b file = illumos-utils/src/zone.rs:316
4722024-10-30T17:47:33.260ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 file = illumos-utils/src/zone.rs:316
4732024-10-30T17:47:33.272ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58 file = illumos-utils/src/zone.rs:354
4742024-10-30T17:47:33.276ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b file = illumos-utils/src/zone.rs:354
4752024-10-30T17:47:33.280ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 file = illumos-utils/src/zone.rs:354
4762024-10-30T17:47:33.326ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/probe_manager.rs:230 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4772024-10-30T17:47:34.327ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/probe_manager.rs:230 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
4782024-10-30T17:47:34.839ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-10-30T17:47:34.839ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4802024-10-30T17:47:34.839ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
4812024-10-30T17:47:35.322ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4822024-10-30T17:47:35.322ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4832024-10-30T17:47:35.801ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4842024-10-30T17:47:35.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 65.048304ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
4852024-10-30T17:47:35.807ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
4862024-10-30T17:47:35.810ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4872024-10-30T17:47:35.810ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
4882024-10-30T17:47:35.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 57.985784ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
4892024-10-30T17:47:35.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 70.011299ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
4902024-10-30T17:47:35.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 53.994497ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
4912024-10-30T17:47:35.947ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 283.019356ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
4922024-10-30T17:47:36.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 71.955222ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
4932024-10-30T17:47:36.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 91.180596ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
4942024-10-30T17:47:36.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 133.697213ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
4952024-10-30T17:47:36.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 361.086856ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
4962024-10-30T17:47:36.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 281.497032ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
4972024-10-30T17:47:36.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 389.141008ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
4982024-10-30T17:47:36.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 507.368596ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
4992024-10-30T17:47:36.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 555.20255ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
5002024-10-30T17:47:36.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 1.185779664s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
5012024-10-30T17:47:37.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 558.215903ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5022024-10-30T17:47:37.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 1.236634752s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
5032024-10-30T17:47:37.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 588.975432ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5042024-10-30T17:47:38.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 976.220451ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
5052024-10-30T17:47:38.058ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16925165195463020634 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
5062024-10-30T17:47:38.058ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 13094888867234479499 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
5072024-10-30T17:47:38.184ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4253
5082024-10-30T17:47:38.185ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4263 gateway = fd00:1122:3344:101::1
5092024-10-30T17:47:38.192ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4342
5102024-10-30T17:47:38.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 582.683955ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5112024-10-30T17:47:38.300ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4390
5122024-10-30T17:47:38.300ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4417
5132024-10-30T17:47:38.337ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4430
5142024-10-30T17:47:38.337ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4294
5152024-10-30T17:47:38.391ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4336
5162024-10-30T17:47:38.391ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4397 rack_subnet = fd00:1122:3344::
5172024-10-30T17:47:38.410ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4408
5182024-10-30T17:47:38.410ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4450
5192024-10-30T17:47:38.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 995.241032ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
5202024-10-30T17:47:38.479ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4482
5212024-10-30T17:47:38.479ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4488
5222024-10-30T17:47:38.563ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4529
5232024-10-30T17:47:38.563ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:370
5242024-10-30T17:47:38.595ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/297da6c0-3dc5-4f20-9a57-48a339550273) file = sled-agent/src/nexus.rs:527 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
5252024-10-30T17:47:38.595ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 41.182975ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5262024-10-30T17:47:38.638ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 114.166632ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5272024-10-30T17:47:38.753ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 192.566424ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id) file = sled-agent/src/bootstrap/early_networking.rs:389
5282024-10-30T17:47:38.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 755.051997ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5292024-10-30T17:47:38.946ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:47833
5302024-10-30T17:47:38.947ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:47833 req_id = bf30f354-1ff8-4e74-abba-65656d725b8a response_code = 200 uri = /sled-identifiers
5312024-10-30T17:47:38.947ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:418
5322024-10-30T17:47:38.979ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:686
5332024-10-30T17:47:38.980ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:724
5342024-10-30T17:47:38.980ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:727 uptime = 0
5352024-10-30T17:47:38.980ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:440
5362024-10-30T17:47:39.020ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
5372024-10-30T17:47:39.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 993.833836ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
5382024-10-30T17:47:39.070ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5392024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5402024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5412024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5422024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5432024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5442024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5452024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5462024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5472024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5482024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5492024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5502024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5512024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5522024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5532024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5542024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5552024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5562024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5572024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5582024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5592024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5602024-10-30T17:47:39.070ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5612024-10-30T17:47:39.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b") failed: Property not found. retry in 1.217048794s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
5622024-10-30T17:47:39.479ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5561917105458027348 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
5632024-10-30T17:47:39.479ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 13283554707908944394 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
5642024-10-30T17:47:39.646ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 1.069751415s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5652024-10-30T17:47:40.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 842.523308ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
5662024-10-30T17:47:40.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63") failed: Property not found. retry in 803.749889ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5672024-10-30T17:47:40.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58") failed: Property not found. retry in 1.221673331s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
5682024-10-30T17:47:42.122ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_internal_dns_3b08df41-4650-4670-af37-80add36afc58
5692024-10-30T17:47:42.122ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_internal_dns_81ce448e-0e36-4fdf-a85d-a98a9892709b
5702024-10-30T17:47:42.122ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_internal_dns_6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63
5712024-10-30T17:47:42.122ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5722024-10-30T17:47:42.123ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5732024-10-30T17:47:42.123ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9144232 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:42875 req_id = f6a3d075-3928-41ae-892e-aa681eb0d23e response_code = 204 uri = /omicron-zones
5742024-10-30T17:47:42.125ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5752024-10-30T17:47:42.172ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5762024-10-30T17:47:42.220ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5772024-10-30T17:47:42.273ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5782024-10-30T17:47:42.273ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5792024-10-30T17:47:42.305ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5802024-10-30T17:47:42.306ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5812024-10-30T17:47:42.306ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5822024-10-30T17:47:42.335ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5832024-10-30T17:47:42.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41622
5842024-10-30T17:47:42.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5852024-10-30T17:47:42.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5862024-10-30T17:47:42.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5872024-10-30T17:47:42.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5882024-10-30T17:47:42.374ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5892024-10-30T17:47:42.374ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5902024-10-30T17:47:42.391ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5912024-10-30T17:47:42.391ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
5922024-10-30T17:47:42.392ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
5932024-10-30T17:47:42.392ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5942024-10-30T17:47:42.403ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
5952024-10-30T17:47:42.435ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
5962024-10-30T17:47:42.435ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
5972024-10-30T17:47:42.435ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
5982024-10-30T17:47:42.464ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 247, 234]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5992024-10-30T17:47:42.464ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1260 zone_type = boundary_ntp
6002024-10-30T17:47:42.465ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
6012024-10-30T17:47:42.465ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
6022024-10-30T17:47:42.465ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
6032024-10-30T17:47:42.466ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6042024-10-30T17:47:42.489ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4 file = illumos-utils/src/zone.rs:316
6052024-10-30T17:47:42.507ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4 file = illumos-utils/src/zone.rs:354
6062024-10-30T17:47:43.130ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6072024-10-30T17:47:43.131ZINFOSledAgent (ServiceManager): Booting oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6082024-10-30T17:47:43.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 71.553614ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6092024-10-30T17:47:43.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 65.032977ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6102024-10-30T17:47:43.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 287.855547ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6112024-10-30T17:47:43.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 559.103343ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6122024-10-30T17:47:44.317ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 1.001338197s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6132024-10-30T17:47:45.328ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 1.372435002s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6142024-10-30T17:47:46.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 795.825103ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6152024-10-30T17:47:47.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4") failed: Property not found. retry in 963.50721ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6162024-10-30T17:47:48.059ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 16925165195463020634 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6172024-10-30T17:47:48.059ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 13094888867234479499 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6182024-10-30T17:47:48.499ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4
6192024-10-30T17:47:48.499ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6202024-10-30T17:47:48.499ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6212024-10-30T17:47:48.500ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6164702 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41622 req_id = 54d8ee6d-b111-4b41-a2c5-51fd087f8cb3 response_code = 204 uri = /omicron-zones
6222024-10-30T17:47:48.502ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6232024-10-30T17:47:48.533ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6242024-10-30T17:47:48.534ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39747
6252024-10-30T17:47:48.543ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3861
6262024-10-30T17:47:48.543ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 9043 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39747 req_id = f4fbe02a-e82d-4fbb-ad8c-8c7cd5637fd4 response_code = 500 uri = /timesync
6272024-10-30T17:47:48.544ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6282024-10-30T17:47:48.668ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6292024-10-30T17:47:48.668ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:39754
6302024-10-30T17:47:49.481ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5561917105458027348 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6312024-10-30T17:47:49.481ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 13283554707908944394 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6322024-10-30T17:47:50.717ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 6176001474828307313 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6332024-10-30T17:47:51.550ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4541436478240252351 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6342024-10-30T17:47:52.125ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 12067437029835989567 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6352024-10-30T17:47:52.677ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3861
6362024-10-30T17:47:52.677ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4009578 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:39754 req_id = 79255d0d-d191-44df-9253-ab664a42eb01 response_code = 500 uri = /timesync
6372024-10-30T17:47:52.680ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6382024-10-30T17:47:52.815ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6392024-10-30T17:47:52.815ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44616
6402024-10-30T17:47:52.824ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3861
6412024-10-30T17:47:52.824ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8439 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44616 req_id = 9c4096f1-ef49-4c69-bd55-1389672b7014 response_code = 500 uri = /timesync
6422024-10-30T17:47:52.827ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6432024-10-30T17:47:53.014ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6442024-10-30T17:47:53.014ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50445
6452024-10-30T17:47:57.189ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_74b6f00a-1e19-4023-8ade-19cb01eb4be4': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3861
6462024-10-30T17:47:57.189ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008248 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50445 req_id = 96c98fdd-cb3d-48d0-b194-64f1bac1e467 response_code = 500 uri = /timesync
6472024-10-30T17:47:57.192ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6482024-10-30T17:47:57.381ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6492024-10-30T17:47:57.381ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64045
6502024-10-30T17:47:57.391ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9387 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64045 req_id = d5aa0b15-14ef-438e-beb9-8541a44ac889 response_code = 200 uri = /timesync
6512024-10-30T17:47:57.392ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6522024-10-30T17:47:57.392ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6532024-10-30T17:47:57.670ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6542024-10-30T17:47:57.670ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45535
6552024-10-30T17:47:57.679ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8637 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45535 req_id = e4523841-515f-4c7f-989a-e433de17e42f response_code = 200 uri = /timesync
6562024-10-30T17:47:57.682ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6572024-10-30T17:47:57.682ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6582024-10-30T17:47:58.016ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6592024-10-30T17:47:58.016ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47925
6602024-10-30T17:47:58.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 21394 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47925 req_id = ccda1479-327c-44a5-ad92-6293486becf1 response_code = 200 uri = /timesync
6612024-10-30T17:47:58.042ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 3330918968, ip_addr: 198.137.202.56, stratum: 3, ref_time: 1730310477.9173894, correction: -0.013613228 } file = sled-agent/src/rack_setup/service.rs:660
6622024-10-30T17:47:58.042ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1298
6632024-10-30T17:47:58.072ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6642024-10-30T17:47:58.073ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33136
6652024-10-30T17:47:58.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6662024-10-30T17:47:58.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6672024-10-30T17:47:58.093ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6682024-10-30T17:47:58.093ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6692024-10-30T17:47:58.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6702024-10-30T17:47:58.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6712024-10-30T17:47:58.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6722024-10-30T17:47:58.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6732024-10-30T17:47:58.180ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6742024-10-30T17:47:58.180ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6752024-10-30T17:47:58.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6762024-10-30T17:47:58.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6772024-10-30T17:47:58.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6782024-10-30T17:47:58.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6792024-10-30T17:47:58.338ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6802024-10-30T17:47:58.338ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
6812024-10-30T17:47:58.389ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6822024-10-30T17:47:58.389ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
6832024-10-30T17:47:58.389ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
6842024-10-30T17:47:58.389ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6852024-10-30T17:47:58.420ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
6862024-10-30T17:47:58.420ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
6872024-10-30T17:47:58.420ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
6882024-10-30T17:47:58.420ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
6892024-10-30T17:47:58.420ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
6902024-10-30T17:47:58.420ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized), ("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 348222 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = 2e7c6398-a965-42a6-bc7c-f1381e1d5ba3 response_code = 500 uri = /omicron-zones
6912024-10-30T17:47:58.420ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2e7c6398-a965-42a6-bc7c-f1381e1d5ba3", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2e7c6398-a965-42a6-bc7c-f1381e1d5ba3" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 94.00625ms sled_address = [fd00:1122:3344:101::1]:12345
6922024-10-30T17:47:58.515ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6932024-10-30T17:47:58.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6942024-10-30T17:47:58.516ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6952024-10-30T17:47:58.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6962024-10-30T17:47:58.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6972024-10-30T17:47:58.540ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 157.020522ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
6982024-10-30T17:47:58.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6992024-10-30T17:47:58.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7002024-10-30T17:47:58.571ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7012024-10-30T17:47:58.571ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7022024-10-30T17:47:58.589ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7032024-10-30T17:47:58.589ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7042024-10-30T17:47:58.606ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7052024-10-30T17:47:58.606ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7062024-10-30T17:47:58.624ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7072024-10-30T17:47:58.624ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7082024-10-30T17:47:58.641ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7092024-10-30T17:47:58.641ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
7102024-10-30T17:47:58.659ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7112024-10-30T17:47:58.659ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
7122024-10-30T17:47:58.659ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
7132024-10-30T17:47:58.659ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7142024-10-30T17:47:58.693ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
7152024-10-30T17:47:58.693ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
7162024-10-30T17:47:58.693ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
7172024-10-30T17:47:58.693ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
7182024-10-30T17:47:58.693ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
7192024-10-30T17:47:58.693ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized), ("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 178458 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = 567b21b6-dd25-4bed-80c1-e2197aa3f658 response_code = 500 uri = /omicron-zones
7202024-10-30T17:47:58.694ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "567b21b6-dd25-4bed-80c1-e2197aa3f658", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "567b21b6-dd25-4bed-80c1-e2197aa3f658" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 80.308816ms sled_address = [fd00:1122:3344:101::1]:12345
7212024-10-30T17:47:58.774ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7222024-10-30T17:47:58.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7232024-10-30T17:47:58.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7242024-10-30T17:47:58.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7252024-10-30T17:47:58.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7262024-10-30T17:47:58.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7272024-10-30T17:47:58.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7282024-10-30T17:47:58.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7292024-10-30T17:47:58.831ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7302024-10-30T17:47:58.849ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7312024-10-30T17:47:58.849ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7322024-10-30T17:47:58.866ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7332024-10-30T17:47:58.866ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7342024-10-30T17:47:58.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7352024-10-30T17:47:58.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7362024-10-30T17:47:58.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7372024-10-30T17:47:58.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
7382024-10-30T17:47:58.918ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7392024-10-30T17:47:58.918ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
7402024-10-30T17:47:58.919ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
7412024-10-30T17:47:58.919ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7422024-10-30T17:47:58.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
7432024-10-30T17:47:58.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
7442024-10-30T17:47:58.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
7452024-10-30T17:47:58.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
7462024-10-30T17:47:58.950ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
7472024-10-30T17:47:58.950ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized), ("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 176225 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = 4ceea7dd-9e9d-45dc-96d4-2be725dca828 response_code = 500 uri = /omicron-zones
7482024-10-30T17:47:58.950ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4ceea7dd-9e9d-45dc-96d4-2be725dca828", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4ceea7dd-9e9d-45dc-96d4-2be725dca828" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 160.955338ms sled_address = [fd00:1122:3344:101::1]:12345
7492024-10-30T17:47:59.111ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7502024-10-30T17:47:59.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7512024-10-30T17:47:59.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7522024-10-30T17:47:59.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7532024-10-30T17:47:59.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7542024-10-30T17:47:59.149ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7552024-10-30T17:47:59.149ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7562024-10-30T17:47:59.167ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7572024-10-30T17:47:59.167ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7582024-10-30T17:47:59.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7592024-10-30T17:47:59.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7602024-10-30T17:47:59.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7612024-10-30T17:47:59.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7622024-10-30T17:47:59.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7632024-10-30T17:47:59.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7642024-10-30T17:47:59.238ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7652024-10-30T17:47:59.238ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
7662024-10-30T17:47:59.256ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7672024-10-30T17:47:59.256ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
7682024-10-30T17:47:59.256ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
7692024-10-30T17:47:59.256ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7702024-10-30T17:47:59.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
7712024-10-30T17:47:59.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
7722024-10-30T17:47:59.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
7732024-10-30T17:47:59.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
7742024-10-30T17:47:59.288ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
7752024-10-30T17:47:59.288ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized), ("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 177576 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = 6b2dc541-a910-4fee-a193-eac27227b539 response_code = 500 uri = /omicron-zones
7762024-10-30T17:47:59.289ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6b2dc541-a910-4fee-a193-eac27227b539", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6b2dc541-a910-4fee-a193-eac27227b539" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 111.107553ms sled_address = [fd00:1122:3344:101::1]:12345
7772024-10-30T17:47:59.400ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7782024-10-30T17:47:59.401ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7792024-10-30T17:47:59.401ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7802024-10-30T17:47:59.419ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7812024-10-30T17:47:59.419ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7822024-10-30T17:47:59.438ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7832024-10-30T17:47:59.438ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7842024-10-30T17:47:59.456ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7852024-10-30T17:47:59.456ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7862024-10-30T17:47:59.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7872024-10-30T17:47:59.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7882024-10-30T17:47:59.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7892024-10-30T17:47:59.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7902024-10-30T17:47:59.509ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7912024-10-30T17:47:59.509ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7922024-10-30T17:47:59.527ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7932024-10-30T17:47:59.527ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
7942024-10-30T17:47:59.544ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7952024-10-30T17:47:59.544ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
7962024-10-30T17:47:59.545ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
7972024-10-30T17:47:59.545ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7982024-10-30T17:47:59.577ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
7992024-10-30T17:47:59.577ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
8002024-10-30T17:47:59.577ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
8012024-10-30T17:47:59.577ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
8022024-10-30T17:47:59.577ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
8032024-10-30T17:47:59.577ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 177095 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = ee771e06-1755-48d6-ad85-1e2241168956 response_code = 500 uri = /omicron-zones
8042024-10-30T17:47:59.577ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ee771e06-1755-48d6-ad85-1e2241168956", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ee771e06-1755-48d6-ad85-1e2241168956" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 167.602675ms sled_address = [fd00:1122:3344:101::1]:12345
8052024-10-30T17:47:59.746ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8062024-10-30T17:47:59.746ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8072024-10-30T17:47:59.746ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8082024-10-30T17:47:59.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8092024-10-30T17:47:59.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8102024-10-30T17:47:59.783ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8112024-10-30T17:47:59.783ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8122024-10-30T17:47:59.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8132024-10-30T17:47:59.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8142024-10-30T17:47:59.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8152024-10-30T17:47:59.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8162024-10-30T17:47:59.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8172024-10-30T17:47:59.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8182024-10-30T17:47:59.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8192024-10-30T17:47:59.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8202024-10-30T17:47:59.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8212024-10-30T17:47:59.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
8222024-10-30T17:47:59.890ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8232024-10-30T17:47:59.890ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
8242024-10-30T17:47:59.890ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
8252024-10-30T17:47:59.890ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8262024-10-30T17:47:59.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
8272024-10-30T17:47:59.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
8282024-10-30T17:47:59.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
8292024-10-30T17:47:59.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
8302024-10-30T17:47:59.922ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
8312024-10-30T17:47:59.922ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 176176 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = 17c2e247-7438-47c2-9132-48af7be21e6a response_code = 500 uri = /omicron-zones
8322024-10-30T17:47:59.922ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "17c2e247-7438-47c2-9132-48af7be21e6a", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "17c2e247-7438-47c2-9132-48af7be21e6a" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 134.067396ms sled_address = [fd00:1122:3344:101::1]:12345
8332024-10-30T17:48:00.057ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8342024-10-30T17:48:00.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8352024-10-30T17:48:00.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8362024-10-30T17:48:00.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8372024-10-30T17:48:00.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8382024-10-30T17:48:00.097ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8392024-10-30T17:48:00.097ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8402024-10-30T17:48:00.115ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8412024-10-30T17:48:00.115ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8422024-10-30T17:48:00.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8432024-10-30T17:48:00.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8442024-10-30T17:48:00.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8452024-10-30T17:48:00.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8462024-10-30T17:48:00.169ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8472024-10-30T17:48:00.169ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8482024-10-30T17:48:00.187ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8492024-10-30T17:48:00.187ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
8502024-10-30T17:48:00.205ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8512024-10-30T17:48:00.205ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
8522024-10-30T17:48:00.205ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
8532024-10-30T17:48:00.205ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8542024-10-30T17:48:00.236ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
8552024-10-30T17:48:00.236ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
8562024-10-30T17:48:00.236ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
8572024-10-30T17:48:00.236ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
8582024-10-30T17:48:00.236ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
8592024-10-30T17:48:00.236ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized), ("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 179509 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = c6c443f0-2c28-467f-ac26-df5249e85b7f response_code = 500 uri = /omicron-zones
8602024-10-30T17:48:00.236ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c6c443f0-2c28-467f-ac26-df5249e85b7f", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c6c443f0-2c28-467f-ac26-df5249e85b7f" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 276.915498ms sled_address = [fd00:1122:3344:101::1]:12345
8612024-10-30T17:48:00.514ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8622024-10-30T17:48:00.514ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8632024-10-30T17:48:00.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8642024-10-30T17:48:00.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8652024-10-30T17:48:00.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8662024-10-30T17:48:00.550ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8672024-10-30T17:48:00.550ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8682024-10-30T17:48:00.567ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8692024-10-30T17:48:00.567ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8702024-10-30T17:48:00.585ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8712024-10-30T17:48:00.585ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8722024-10-30T17:48:00.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8732024-10-30T17:48:00.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8742024-10-30T17:48:00.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8752024-10-30T17:48:00.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8762024-10-30T17:48:00.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8772024-10-30T17:48:00.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
8782024-10-30T17:48:00.656ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8792024-10-30T17:48:00.656ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
8802024-10-30T17:48:00.656ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
8812024-10-30T17:48:00.656ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8822024-10-30T17:48:00.689ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
8832024-10-30T17:48:00.689ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
8842024-10-30T17:48:00.689ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
8852024-10-30T17:48:00.689ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
8862024-10-30T17:48:00.689ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3276 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
8872024-10-30T17:48:00.689ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62", TimeNotSynchronized), ("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392", TimeNotSynchronized), ("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910", TimeNotSynchronized), ("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26", TimeNotSynchronized), ("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 175260 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = 8ad5da9d-13e8-4705-8e94-66e489ab8843 response_code = 500 uri = /omicron-zones
8882024-10-30T17:48:00.689ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8ad5da9d-13e8-4705-8e94-66e489ab8843", "content-length": "124", "date": "Wed, 30 Oct 2024 17:47:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8ad5da9d-13e8-4705-8e94-66e489ab8843" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 229.318866ms sled_address = [fd00:1122:3344:101::1]:12345
8892024-10-30T17:48:00.919ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8902024-10-30T17:48:00.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8912024-10-30T17:48:00.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8922024-10-30T17:48:00.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8932024-10-30T17:48:00.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8942024-10-30T17:48:00.956ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8952024-10-30T17:48:00.956ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8962024-10-30T17:48:00.974ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8972024-10-30T17:48:00.974ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8982024-10-30T17:48:00.993ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8992024-10-30T17:48:00.993ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9002024-10-30T17:48:01.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9012024-10-30T17:48:01.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9022024-10-30T17:48:01.029ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9032024-10-30T17:48:01.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9042024-10-30T17:48:01.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9052024-10-30T17:48:01.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
9062024-10-30T17:48:01.065ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9072024-10-30T17:48:01.065ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
9082024-10-30T17:48:01.065ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
9092024-10-30T17:48:01.065ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9102024-10-30T17:48:01.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9112024-10-30T17:48:01.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9122024-10-30T17:48:01.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9132024-10-30T17:48:01.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9142024-10-30T17:48:01.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9152024-10-30T17:48:01.238ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62 file = illumos-utils/src/zone.rs:316
9162024-10-30T17:48:01.242ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392 file = illumos-utils/src/zone.rs:316
9172024-10-30T17:48:01.246ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db file = illumos-utils/src/zone.rs:316
9182024-10-30T17:48:01.250ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910 file = illumos-utils/src/zone.rs:316
9192024-10-30T17:48:01.254ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 file = illumos-utils/src/zone.rs:316
9202024-10-30T17:48:01.259ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62 file = illumos-utils/src/zone.rs:354
9212024-10-30T17:48:01.264ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392 file = illumos-utils/src/zone.rs:354
9222024-10-30T17:48:01.268ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db file = illumos-utils/src/zone.rs:354
9232024-10-30T17:48:01.272ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910 file = illumos-utils/src/zone.rs:354
9242024-10-30T17:48:01.277ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 file = illumos-utils/src/zone.rs:354
9252024-10-30T17:48:04.125ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='99f16332-1177-4807-8f95-47cd34b7d7db'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9262024-10-30T17:48:04.125ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9272024-10-30T17:48:04.140ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='10f9ea9a-ebab-4205-a225-0cb97f84eb62'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9282024-10-30T17:48:04.140ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9292024-10-30T17:48:04.203ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='c17b59dc-32ef-41d3-b98f-354d10ced910'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9302024-10-30T17:48:04.203ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9312024-10-30T17:48:04.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 58.407019ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9322024-10-30T17:48:04.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 68.914371ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9332024-10-30T17:48:04.289ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='71ecea3f-6c94-4ddb-b17c-44303894e392'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9342024-10-30T17:48:04.289ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9352024-10-30T17:48:04.302ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='dfb3cbb2-78d7-42e2-a8e4-f077a6736b26'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9362024-10-30T17:48:04.302ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9372024-10-30T17:48:04.337ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 45.695387ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9382024-10-30T17:48:04.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 89.658976ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9392024-10-30T17:48:04.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 122.276436ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9402024-10-30T17:48:04.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 133.857531ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9412024-10-30T17:48:04.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 181.738428ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9422024-10-30T17:48:04.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 123.040569ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9432024-10-30T17:48:04.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 30.381756ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9442024-10-30T17:48:04.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 229.247718ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9452024-10-30T17:48:04.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 67.009181ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9462024-10-30T17:48:04.680ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 53.922954ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9472024-10-30T17:48:04.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 550.958744ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9482024-10-30T17:48:04.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 174.866796ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9492024-10-30T17:48:05.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 388.973871ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9502024-10-30T17:48:05.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 340.499297ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9512024-10-30T17:48:05.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 149.559145ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9522024-10-30T17:48:05.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 263.435383ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9532024-10-30T17:48:05.664ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 596.715202ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9542024-10-30T17:48:05.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 957.841007ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9552024-10-30T17:48:05.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 1.133594121s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9562024-10-30T17:48:06.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 194.678724ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9572024-10-30T17:48:06.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 604.966229ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9582024-10-30T17:48:06.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 247.885454ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9592024-10-30T17:48:07.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 605.450957ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9602024-10-30T17:48:07.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 878.294671ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9612024-10-30T17:48:07.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 969.437027ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9622024-10-30T17:48:07.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 1.340518627s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9632024-10-30T17:48:07.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 822.58771ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9642024-10-30T17:48:07.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 590.451157ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9652024-10-30T17:48:07.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 642.30728ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9662024-10-30T17:48:08.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.311888657s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9672024-10-30T17:48:08.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 1.306619571s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9682024-10-30T17:48:08.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 1.128786954s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9692024-10-30T17:48:08.437ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 1.035403294s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9702024-10-30T17:48:08.554ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46159
9712024-10-30T17:48:08.554ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46159 req_id = 2c0b9679-8160-4ec5-9aee-be21d35bb5a2 response_code = 200 uri = /baseboard
9722024-10-30T17:48:08.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 748.394625ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9732024-10-30T17:48:09.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 787.104056ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9742024-10-30T17:48:09.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 778.124294ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9752024-10-30T17:48:09.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 777.006298ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9762024-10-30T17:48:09.383ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 551.413936ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9772024-10-30T17:48:09.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 608.398695ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9782024-10-30T17:48:09.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 809.394806ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9792024-10-30T17:48:10.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 977.240397ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9802024-10-30T17:48:10.129ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 1.080892145s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9812024-10-30T17:48:10.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.447258409s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9822024-10-30T17:48:10.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 908.343522ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9832024-10-30T17:48:10.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 1.215852205s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9842024-10-30T17:48:11.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 1.272007502s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9852024-10-30T17:48:11.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 1.225637367s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9862024-10-30T17:48:11.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 558.029025ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9872024-10-30T17:48:11.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 848.85897ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9882024-10-30T17:48:11.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 556.798854ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9892024-10-30T17:48:12.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 975.904438ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9902024-10-30T17:48:12.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 677.177881ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9912024-10-30T17:48:12.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 1.130165809s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9922024-10-30T17:48:12.358ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 1.120055859s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9932024-10-30T17:48:12.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.328890162s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
9942024-10-30T17:48:12.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 848.673286ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
9952024-10-30T17:48:13.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910") failed: Property not found. retry in 1.166656426s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
9962024-10-30T17:48:13.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62") failed: Property not found. retry in 911.153598ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
9972024-10-30T17:48:13.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392") failed: Property not found. retry in 884.507216ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
9982024-10-30T17:48:13.720ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 499.749459ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
9992024-10-30T17:48:13.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.296462038s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10002024-10-30T17:48:13.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 588.106051ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10012024-10-30T17:48:14.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 976.921753ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10022024-10-30T17:48:15.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.217740008s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10032024-10-30T17:48:15.430ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 689.44654ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10042024-10-30T17:48:16.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 829.89725ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10052024-10-30T17:48:16.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.070803007s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10062024-10-30T17:48:17.482ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 1.350565455s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10072024-10-30T17:48:17.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 747.549581ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10082024-10-30T17:48:18.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 909.421473ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10092024-10-30T17:48:18.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 920.148191ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10102024-10-30T17:48:19.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.156590482s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10112024-10-30T17:48:19.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 1.128511521s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10122024-10-30T17:48:20.584ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.377771705s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10132024-10-30T17:48:20.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 974.996932ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10142024-10-30T17:48:21.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 1.197057929s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10152024-10-30T17:48:21.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 1.436590663s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10162024-10-30T17:48:23.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26") failed: Property not found. retry in 676.03195ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10172024-10-30T17:48:23.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db") failed: Property not found. retry in 770.281802ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10182024-10-30T17:48:24.208ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_cockroachdb_10f9ea9a-ebab-4205-a225-0cb97f84eb62
10192024-10-30T17:48:24.208ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_cockroachdb_71ecea3f-6c94-4ddb-b17c-44303894e392
10202024-10-30T17:48:24.208ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_cockroachdb_99f16332-1177-4807-8f95-47cd34b7d7db
10212024-10-30T17:48:24.208ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_cockroachdb_c17b59dc-32ef-41d3-b98f-354d10ced910
10222024-10-30T17:48:24.208ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_cockroachdb_dfb3cbb2-78d7-42e2-a8e4-f077a6736b26
10232024-10-30T17:48:24.208ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
10242024-10-30T17:48:24.209ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
10252024-10-30T17:48:24.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 23296918 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33136 req_id = e9910e1a-0e67-4f8f-9ce2-0fa111c18f06 response_code = 204 uri = /omicron-zones
10262024-10-30T17:48:24.243ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54000
10272024-10-30T17:48:24.274ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::4]:32221 file = sled-agent/src/services.rs:3704
10282024-10-30T17:48:24.461ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3722
10292024-10-30T17:48:29.917ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3745
10302024-10-30T17:48:29.917ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 5673763 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:54000 req_id = 25393f90-68ae-4020-b670-939c40e7a35a response_code = 204 uri = /cockroachdb
10312024-10-30T17:48:29.922ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 589.647387ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
10322024-10-30T17:48:29.954ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 3b08df41-4650-4670-af37-80add36afc58 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 81ce448e-0e36-4fdf-a85d-a98a9892709b (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 2b82bbdf-eb96-4141-b2cf-19f5fb5b406c, kind: Service { id: 74b6f00a-1e19-4023-8ade-19cb01eb4be4 }, name: Name("ntp-74b6f00a-1e19-4023-8ade-19cb01eb4be4"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 247, 234])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 71ecea3f-6c94-4ddb-b17c-44303894e392 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: c17b59dc-32ef-41d3-b98f-354d10ced910 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26 (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 99f16332-1177-4807-8f95-47cd34b7d7db (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }, OmicronZoneConfig { id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: 25610ba7-70b1-4917-a427-f5070892a54f, kind: Service { id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb }, name: Name("external-dns-185f90b2-80bf-479a-b8b0-ec1f3c02a2fb"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 105])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 0817b15f-d0fb-4e3e-8619-51d448476a89, kind: Service { id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 }, name: Name("external-dns-ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 59])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 084cb44c-d8f2-4ea3-8498-21306b882ab8 (service), filesystem_pool: Some(ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: e6a0dd9f-e087-4f2a-b710-699e007ce774, kind: Service { id: 084cb44c-d8f2-4ea3-8498-21306b882ab8 }, name: Name("nexus-084cb44c-d8f2-4ea3-8498-21306b882ab8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: ee372901-c3d9-45f0-8c4f-5c4702a28b4d (service), filesystem_pool: Some(ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 6734898c-60d3-45c1-8b48-7163e5e7e5d7, kind: Service { id: ee372901-c3d9-45f0-8c4f-5c4702a28b4d }, name: Name("nexus-ee372901-c3d9-45f0-8c4f-5c4702a28b4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 33])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: c24ae96c-052c-4566-8772-4d7cffb20cb5 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: a90bbb13-856b-46a6-8e82-4b9742f6c506, kind: Service { id: c24ae96c-052c-4566-8772-4d7cffb20cb5 }, name: Name("nexus-c24ae96c-052c-4566-8772-4d7cffb20cb5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 7f02d345-3366-4b62-9cd8-995f8742dd0c (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 366b95de-05cf-43cb-8b58-0d4b6e1b2574 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: f490660a-0116-4f0f-b1bc-faa5f011ce36 (service), filesystem_pool: Some(ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: eaea67ce-5679-46bf-b0bf-4a021e5bbec3 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: ea535df2-fd38-4652-9e6f-02fe39690a9c (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346 (service), filesystem_pool: Some(ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 7180a284-fbab-4f51-8bb9-97f62b818245 (service), filesystem_pool: Some(ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External } } } }, OmicronZoneConfig { id: 04ada316-c920-4e4a-9d41-4c467b02ec90 (service), filesystem_pool: Some(ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0be13225-a78b-4173-9c0a-759ebd323d2d (service), filesystem_pool: Some(ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: 0206d1cc-2f17-4c82-9c67-f4c41faa893d (service), filesystem_pool: Some(ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External } } } }, OmicronZoneConfig { id: 34b851f8-bee3-4493-8196-84205c8c2127 (service), filesystem_pool: Some(ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External } } } }, OmicronZoneConfig { id: 530c539e-4272-4675-882c-e4d1d574c146 (service), filesystem_pool: Some(ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External } } } }, OmicronZoneConfig { id: a9aa814a-015b-4f38-8445-d17789da9b82 (service), filesystem_pool: Some(ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External } } } }, OmicronZoneConfig { id: c61c51d9-a3b2-4ccb-ac02-b78ef7496910 (service), filesystem_pool: Some(ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
10332024-10-30T17:48:29.954ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59677
10342024-10-30T17:48:29.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10352024-10-30T17:48:29.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 3b08df41-4650-4670-af37-80add36afc58, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10362024-10-30T17:48:29.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10372024-10-30T17:48:29.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 6f3eb7b7-d63d-46e2-9002-9eb0cdaccc63, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10382024-10-30T17:48:29.999ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10392024-10-30T17:48:29.999ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 81ce448e-0e36-4fdf-a85d-a98a9892709b, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10402024-10-30T17:48:30.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10412024-10-30T17:48:30.020ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 71ecea3f-6c94-4ddb-b17c-44303894e392, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10422024-10-30T17:48:30.043ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10432024-10-30T17:48:30.043ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 10f9ea9a-ebab-4205-a225-0cb97f84eb62, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10442024-10-30T17:48:30.063ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10452024-10-30T17:48:30.063ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c17b59dc-32ef-41d3-b98f-354d10ced910, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10462024-10-30T17:48:30.081ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10472024-10-30T17:48:30.081ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: dfb3cbb2-78d7-42e2-a8e4-f077a6736b26, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10482024-10-30T17:48:30.103ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10492024-10-30T17:48:30.103ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 99f16332-1177-4807-8f95-47cd34b7d7db, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Cockroach }, responder: ... }
10502024-10-30T17:48:30.122ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10512024-10-30T17:48:30.122ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10522024-10-30T17:48:30.201ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10532024-10-30T17:48:30.201ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10542024-10-30T17:48:30.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 366b95de-05cf-43cb-8b58-0d4b6e1b2574, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
10552024-10-30T17:48:30.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 366b95de-05cf-43cb-8b58-0d4b6e1b2574, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10562024-10-30T17:48:30.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10572024-10-30T17:48:30.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346, dataset_name: DatasetName { pool_name: ZpoolName { id: 5c811413-1d4e-403e-b8bb-331b2268c476 (zpool), kind: External }, kind: Crucible }, responder: ... }
10582024-10-30T17:48:30.447ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 7180a284-fbab-4f51-8bb9-97f62b818245, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10592024-10-30T17:48:30.447ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 7180a284-fbab-4f51-8bb9-97f62b818245, dataset_name: DatasetName { pool_name: ZpoolName { id: 7c337fba-b737-4f90-a327-065b4f3b4476 (zpool), kind: External }, kind: Crucible }, responder: ... }
10602024-10-30T17:48:30.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 04ada316-c920-4e4a-9d41-4c467b02ec90, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10612024-10-30T17:48:30.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 04ada316-c920-4e4a-9d41-4c467b02ec90, dataset_name: DatasetName { pool_name: ZpoolName { id: 46e37abb-39ab-4652-aa9d-61a2079024f1 (zpool), kind: External }, kind: Crucible }, responder: ... }
10622024-10-30T17:48:30.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0be13225-a78b-4173-9c0a-759ebd323d2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10632024-10-30T17:48:30.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0be13225-a78b-4173-9c0a-759ebd323d2d, dataset_name: DatasetName { pool_name: ZpoolName { id: 8b4477e5-c445-4846-a713-78ddac5696a7 (zpool), kind: External }, kind: Crucible }, responder: ... }
10642024-10-30T17:48:30.702ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0206d1cc-2f17-4c82-9c67-f4c41faa893d, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10652024-10-30T17:48:30.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 0206d1cc-2f17-4c82-9c67-f4c41faa893d, dataset_name: DatasetName { pool_name: ZpoolName { id: 09d27db0-c064-443e-8e36-e29bb4d8f58d (zpool), kind: External }, kind: Crucible }, responder: ... }
10662024-10-30T17:48:30.776ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 34b851f8-bee3-4493-8196-84205c8c2127, dataset_name: DatasetName { pool_name: ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10672024-10-30T17:48:30.776ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 34b851f8-bee3-4493-8196-84205c8c2127, dataset_name: DatasetName { pool_name: ZpoolName { id: 9fecadf7-1f9f-4c05-baeb-3ca812eb625c (zpool), kind: External }, kind: Crucible }, responder: ... }
10682024-10-30T17:48:30.842ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 530c539e-4272-4675-882c-e4d1d574c146, dataset_name: DatasetName { pool_name: ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10692024-10-30T17:48:30.842ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 530c539e-4272-4675-882c-e4d1d574c146, dataset_name: DatasetName { pool_name: ZpoolName { id: c84ed810-b8fd-4283-8511-603d43b629a7 (zpool), kind: External }, kind: Crucible }, responder: ... }
10702024-10-30T17:48:30.905ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a9aa814a-015b-4f38-8445-d17789da9b82, dataset_name: DatasetName { pool_name: ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10712024-10-30T17:48:30.905ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: a9aa814a-015b-4f38-8445-d17789da9b82, dataset_name: DatasetName { pool_name: ZpoolName { id: f6c51041-d1f1-4026-854a-17c6531b176e (zpool), kind: External }, kind: Crucible }, responder: ... }
10722024-10-30T17:48:30.965ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c61c51d9-a3b2-4ccb-ac02-b78ef7496910, dataset_name: DatasetName { pool_name: ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
10732024-10-30T17:48:30.965ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: c61c51d9-a3b2-4ccb-ac02-b78ef7496910, dataset_name: DatasetName { pool_name: ZpoolName { id: b47c81a2-215a-4a96-ada9-ce831b19ee43 (zpool), kind: External }, kind: Crucible }, responder: ... }
10742024-10-30T17:48:31.022ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10752024-10-30T17:48:31.022ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
10762024-10-30T17:48:31.022ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
10772024-10-30T17:48:31.022ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10782024-10-30T17:48:31.160ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10792024-10-30T17:48:31.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10802024-10-30T17:48:31.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10812024-10-30T17:48:31.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10822024-10-30T17:48:31.161ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10832024-10-30T17:48:31.183ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10842024-10-30T17:48:31.184ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10852024-10-30T17:48:31.184ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10862024-10-30T17:48:31.184ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10872024-10-30T17:48:31.195ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10882024-10-30T17:48:31.369ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10892024-10-30T17:48:31.384ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10902024-10-30T17:48:31.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10912024-10-30T17:48:31.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10922024-10-30T17:48:31.408ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:264
10932024-10-30T17:48:31.408ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10942024-10-30T17:48:31.408ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10952024-10-30T17:48:31.421ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10962024-10-30T17:48:31.421ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10972024-10-30T17:48:31.421ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574 file = illumos-utils/src/zone.rs:316
10982024-10-30T17:48:31.426ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3 file = illumos-utils/src/zone.rs:316
10992024-10-30T17:48:31.430ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346 file = illumos-utils/src/zone.rs:316
11002024-10-30T17:48:31.436ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146 file = illumos-utils/src/zone.rs:316
11012024-10-30T17:48:31.443ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c file = illumos-utils/src/zone.rs:316
11022024-10-30T17:48:31.450ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d file = illumos-utils/src/zone.rs:316
11032024-10-30T17:48:31.454ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90 file = illumos-utils/src/zone.rs:316
11042024-10-30T17:48:31.459ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82 file = illumos-utils/src/zone.rs:316
11052024-10-30T17:48:31.581ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574 file = illumos-utils/src/zone.rs:354
11062024-10-30T17:48:31.587ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3 file = illumos-utils/src/zone.rs:354
11072024-10-30T17:48:31.592ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346 file = illumos-utils/src/zone.rs:354
11082024-10-30T17:48:31.596ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146 file = illumos-utils/src/zone.rs:354
11092024-10-30T17:48:31.601ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c file = illumos-utils/src/zone.rs:354
11102024-10-30T17:48:31.606ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d file = illumos-utils/src/zone.rs:354
11112024-10-30T17:48:31.610ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90 file = illumos-utils/src/zone.rs:354
11122024-10-30T17:48:31.615ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82 file = illumos-utils/src/zone.rs:354
11132024-10-30T17:48:31.621ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c file = illumos-utils/src/zone.rs:316
11142024-10-30T17:48:31.626ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245 file = illumos-utils/src/zone.rs:316
11152024-10-30T17:48:31.632ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d file = illumos-utils/src/zone.rs:316
11162024-10-30T17:48:31.637ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127 file = illumos-utils/src/zone.rs:316
11172024-10-30T17:48:31.642ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36 file = illumos-utils/src/zone.rs:316
11182024-10-30T17:48:31.648ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910 file = illumos-utils/src/zone.rs:316
11192024-10-30T17:48:31.655ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c file = illumos-utils/src/zone.rs:354
11202024-10-30T17:48:31.661ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245 file = illumos-utils/src/zone.rs:354
11212024-10-30T17:48:31.667ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d file = illumos-utils/src/zone.rs:354
11222024-10-30T17:48:31.671ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127 file = illumos-utils/src/zone.rs:354
11232024-10-30T17:48:31.676ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36 file = illumos-utils/src/zone.rs:354
11242024-10-30T17:48:31.681ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910 file = illumos-utils/src/zone.rs:354
11252024-10-30T17:48:31.718ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
11262024-10-30T17:48:31.751ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
11272024-10-30T17:48:31.787ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
11282024-10-30T17:48:31.831ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
11292024-10-30T17:48:31.885ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:281
11302024-10-30T17:48:31.887ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11312024-10-30T17:48:31.887ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11322024-10-30T17:48:31.934ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 248, 33]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11332024-10-30T17:48:31.934ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1260 zone_type = nexus
11342024-10-30T17:48:31.935ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11352024-10-30T17:48:31.935ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11362024-10-30T17:48:31.985ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 230, 59]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11372024-10-30T17:48:31.985ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1260 zone_type = external_dns
11382024-10-30T17:48:31.987ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11392024-10-30T17:48:31.987ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11402024-10-30T17:48:32.037ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 170, 1]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11412024-10-30T17:48:32.037ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1260 zone_type = nexus
11422024-10-30T17:48:32.038ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11432024-10-30T17:48:32.038ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11442024-10-30T17:48:32.089ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 239, 105]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11452024-10-30T17:48:32.089ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1260 zone_type = external_dns
11462024-10-30T17:48:32.091ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:306
11472024-10-30T17:48:32.091ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:198
11482024-10-30T17:48:32.141ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 130, 23]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11492024-10-30T17:48:32.141ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1260 zone_type = nexus
11502024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
11512024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
11522024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
11532024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
11542024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
11552024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
11562024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
11572024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
11582024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
11592024-10-30T17:48:32.142ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
11602024-10-30T17:48:32.143ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
11612024-10-30T17:48:32.144ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
11622024-10-30T17:48:32.146ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
11632024-10-30T17:48:32.146ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11642024-10-30T17:48:32.146ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11652024-10-30T17:48:32.148ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11662024-10-30T17:48:32.148ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
11672024-10-30T17:48:32.153ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
11682024-10-30T17:48:32.155ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11692024-10-30T17:48:32.155ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11702024-10-30T17:48:32.474ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb file = illumos-utils/src/zone.rs:316
11712024-10-30T17:48:32.492ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8 file = illumos-utils/src/zone.rs:316
11722024-10-30T17:48:32.506ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d file = illumos-utils/src/zone.rs:316
11732024-10-30T17:48:32.535ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 file = illumos-utils/src/zone.rs:316
11742024-10-30T17:48:32.556ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5 file = illumos-utils/src/zone.rs:316
11752024-10-30T17:48:32.577ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb file = illumos-utils/src/zone.rs:354
11762024-10-30T17:48:32.599ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8 file = illumos-utils/src/zone.rs:354
11772024-10-30T17:48:32.614ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d file = illumos-utils/src/zone.rs:354
11782024-10-30T17:48:32.637ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 file = illumos-utils/src/zone.rs:354
11792024-10-30T17:48:32.651ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5 file = illumos-utils/src/zone.rs:354
11802024-10-30T17:48:34.227ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_5c811413-1d4e-403e-b8bb-331b2268c476/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='ab7ba7aa-4a45-407b-bdda-bbfa9b5a84f2'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11812024-10-30T17:48:34.227ZINFOSledAgent (ServiceManager): Booting oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
11822024-10-30T17:48:34.244ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_46e37abb-39ab-4652-aa9d-61a2079024f1/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='6ec73717-7598-42f4-b124-fe6682539dfd'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11832024-10-30T17:48:34.244ZINFOSledAgent (ServiceManager): Booting oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
11842024-10-30T17:48:34.252ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f6c51041-d1f1-4026-854a-17c6531b176e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='03a0310f-4c46-4f6d-ad80-9e4f9ca1c7d3'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11852024-10-30T17:48:34.253ZINFOSledAgent (ServiceManager): Booting oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
11862024-10-30T17:48:34.325ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c84ed810-b8fd-4283-8511-603d43b629a7/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='2c8b3a63-95d2-486b-8f7d-41fb4f2e7ab7'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11872024-10-30T17:48:34.325ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_09d27db0-c064-443e-8e36-e29bb4d8f58d/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='c99627fb-59ca-4420-b105-49c6588edbae'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11882024-10-30T17:48:34.325ZINFOSledAgent (ServiceManager): Booting oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
11892024-10-30T17:48:34.332ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
11902024-10-30T17:48:34.346ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11912024-10-30T17:48:34.346ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
11922024-10-30T17:48:34.599ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_8b4477e5-c445-4846-a713-78ddac5696a7/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='04f5c7f3-1017-4e67-a2d0-a53ed39e76aa'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11932024-10-30T17:48:34.599ZINFOSledAgent (ServiceManager): Booting oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
11942024-10-30T17:48:34.627ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_b47c81a2-215a-4a96-ada9-ce831b19ee43/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='639e17b3-bb8e-4136-96d5-636ff0dabd5d'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11952024-10-30T17:48:34.628ZINFOSledAgent (ServiceManager): Booting oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
11962024-10-30T17:48:34.674ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_9fecadf7-1f9f-4c05-baeb-3ca812eb625c/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='18c8b0a6-16c6-46d7-a414-eb7ef0151591'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
11972024-10-30T17:48:34.675ZINFOSledAgent (ServiceManager): Booting oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
11982024-10-30T17:48:34.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 58.599726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
11992024-10-30T17:48:34.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 27.915739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12002024-10-30T17:48:34.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 66.640752ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12012024-10-30T17:48:34.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 29.484071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12022024-10-30T17:48:34.965ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 68.655436ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12032024-10-30T17:48:34.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 66.286061ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12042024-10-30T17:48:34.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 96.875831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12052024-10-30T17:48:35.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 127.44899ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12062024-10-30T17:48:35.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 47.891428ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12072024-10-30T17:48:35.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 71.810082ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12082024-10-30T17:48:35.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 94.243946ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12092024-10-30T17:48:35.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 58.264372ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12102024-10-30T17:48:35.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 120.07739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12112024-10-30T17:48:35.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 121.51789ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12122024-10-30T17:48:35.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 136.264917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12132024-10-30T17:48:35.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 168.825842ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12142024-10-30T17:48:35.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 113.893272ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12152024-10-30T17:48:35.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 123.969857ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12162024-10-30T17:48:35.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 118.065946ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12172024-10-30T17:48:35.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 231.802757ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12182024-10-30T17:48:35.320ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 121.101092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12192024-10-30T17:48:35.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 202.578268ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12202024-10-30T17:48:35.428ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 156.415655ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12212024-10-30T17:48:35.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 297.772989ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12222024-10-30T17:48:35.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 233.837828ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12232024-10-30T17:48:35.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 223.599286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12242024-10-30T17:48:35.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 266.051174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12252024-10-30T17:48:35.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 255.517335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12262024-10-30T17:48:35.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 593.577667ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12272024-10-30T17:48:35.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 249.739825ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12282024-10-30T17:48:35.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 560.783226ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12292024-10-30T17:48:35.645ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 280.348672ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12302024-10-30T17:48:35.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 445.081152ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12312024-10-30T17:48:35.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 395.611768ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12322024-10-30T17:48:35.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.070926753s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12332024-10-30T17:48:35.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 503.688423ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12342024-10-30T17:48:35.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 373.691842ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12352024-10-30T17:48:35.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 484.292823ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12362024-10-30T17:48:35.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.153471653s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12372024-10-30T17:48:36.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.166460993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12382024-10-30T17:48:36.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 756.346394ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12392024-10-30T17:48:36.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 673.605842ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12402024-10-30T17:48:36.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 826.729734ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12412024-10-30T17:48:36.219ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.057670793s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12422024-10-30T17:48:36.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 571.959176ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12432024-10-30T17:48:36.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 909.131952ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12442024-10-30T17:48:36.836ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 758.906756ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12452024-10-30T17:48:36.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 618.233788ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12462024-10-30T17:48:36.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.236791292s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12472024-10-30T17:48:36.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 696.94428ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12482024-10-30T17:48:37.047ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.136936777s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12492024-10-30T17:48:37.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.38398715s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12502024-10-30T17:48:37.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 500.426169ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12512024-10-30T17:48:37.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 712.197464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12522024-10-30T17:48:37.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 586.726851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12532024-10-30T17:48:37.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.34570728s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12542024-10-30T17:48:37.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.386859744s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12552024-10-30T17:48:37.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.307766977s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12562024-10-30T17:48:37.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.343988492s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12572024-10-30T17:48:37.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 532.037605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12582024-10-30T17:48:38.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 592.129857ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12592024-10-30T17:48:38.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 569.758187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12602024-10-30T17:48:38.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 728.004904ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12612024-10-30T17:48:38.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.198559856s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12622024-10-30T17:48:38.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.014926555s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12632024-10-30T17:48:38.613ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59268
12642024-10-30T17:48:38.618ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 154 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59268 req_id = 1af0e4d9-e4fe-4b46-8c27-9037be4f2440 response_code = 200 uri = /baseboard
12652024-10-30T17:48:38.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.139730921s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12662024-10-30T17:48:38.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.249263843s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12672024-10-30T17:48:38.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.346797088s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12682024-10-30T17:48:38.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.468321903s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12692024-10-30T17:48:38.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 564.869907ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12702024-10-30T17:48:39.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.105608475s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12712024-10-30T17:48:39.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.461139127s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12722024-10-30T17:48:39.546ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 575.151085ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12732024-10-30T17:48:39.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 536.717375ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12742024-10-30T17:48:39.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 625.365359ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12752024-10-30T17:48:39.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.22658971s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12762024-10-30T17:48:39.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 793.295989ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12772024-10-30T17:48:40.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.437789394s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12782024-10-30T17:48:40.127ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 608.297435ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12792024-10-30T17:48:40.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.079572975s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12802024-10-30T17:48:40.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 768.858704ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12812024-10-30T17:48:40.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 860.436425ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12822024-10-30T17:48:40.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.288780699s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
12832024-10-30T17:48:40.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 652.763954ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12842024-10-30T17:48:40.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.322391847s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
12852024-10-30T17:48:40.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 866.113543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
12862024-10-30T17:48:40.991ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12872024-10-30T17:48:40.991ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
12882024-10-30T17:48:41.024ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.171496066s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
12892024-10-30T17:48:41.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 660.252051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
12902024-10-30T17:48:41.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 561.376386ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
12912024-10-30T17:48:41.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 46.24507ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
12922024-10-30T17:48:41.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 753.362178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
12932024-10-30T17:48:41.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 145.410831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
12942024-10-30T17:48:41.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 750.28898ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
12952024-10-30T17:48:41.360ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='7f02d345-3366-4b62-9cd8-995f8742dd0c'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12962024-10-30T17:48:41.360ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
12972024-10-30T17:48:41.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 226.083981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
12982024-10-30T17:48:41.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 545.510221ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
12992024-10-30T17:48:41.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 32.276834ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13002024-10-30T17:48:41.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 78.853371ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13012024-10-30T17:48:41.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 592.66297ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13022024-10-30T17:48:41.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 302.832726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13032024-10-30T17:48:41.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 757.480555ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13042024-10-30T17:48:41.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 121.569139ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13052024-10-30T17:48:41.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.195534352s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13062024-10-30T17:48:41.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 823.536498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13072024-10-30T17:48:41.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 227.275061ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13082024-10-30T17:48:42.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 819.00041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13092024-10-30T17:48:42.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.115198406s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13102024-10-30T17:48:42.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 733.408965ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13112024-10-30T17:48:42.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 906.464111ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13122024-10-30T17:48:42.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 406.68297ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13132024-10-30T17:48:42.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 902.948212ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13142024-10-30T17:48:42.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.143663498s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13152024-10-30T17:48:42.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.312778001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13162024-10-30T17:48:42.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 885.632543ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13172024-10-30T17:48:42.507ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 549.514992ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13182024-10-30T17:48:42.663ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.485604371s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13192024-10-30T17:48:42.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.231202565s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13202024-10-30T17:48:42.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.412251411s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13212024-10-30T17:48:42.953ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 515.534501ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13222024-10-30T17:48:42.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 847.563794ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13232024-10-30T17:48:43.033ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.044592391s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13242024-10-30T17:48:43.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 927.587938ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13252024-10-30T17:48:43.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 638.427786ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13262024-10-30T17:48:43.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.373991s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13272024-10-30T17:48:43.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.276848787s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13282024-10-30T17:48:43.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.283111989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13292024-10-30T17:48:43.602ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.321100921s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13302024-10-30T17:48:43.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.494328969s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13312024-10-30T17:48:43.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 941.781392ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13322024-10-30T17:48:44.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 910.158063ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13332024-10-30T17:48:44.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 754.303123ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13342024-10-30T17:48:44.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.149680018s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13352024-10-30T17:48:44.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.456241s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13362024-10-30T17:48:44.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 538.172401ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13372024-10-30T17:48:44.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.402973836s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13382024-10-30T17:48:44.745ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 843.709071ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13392024-10-30T17:48:44.772ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.005502074s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13402024-10-30T17:48:44.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 524.076491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13412024-10-30T17:48:44.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 749.762028ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13422024-10-30T17:48:44.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 954.699192ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13432024-10-30T17:48:44.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 850.818768ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13442024-10-30T17:48:44.931ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 918.417153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13452024-10-30T17:48:45.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 805.519187ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13462024-10-30T17:48:45.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 954.325384ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13472024-10-30T17:48:45.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 918.397569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13482024-10-30T17:48:45.572ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.68195509s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
13492024-10-30T17:48:45.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 777.479933ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13502024-10-30T17:48:45.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.171616162s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13512024-10-30T17:48:45.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 870.664366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13522024-10-30T17:48:45.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 563.666429ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13532024-10-30T17:48:45.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 611.429175ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13542024-10-30T17:48:45.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.312903259s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13552024-10-30T17:48:45.858ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.473663172s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13562024-10-30T17:48:46.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 952.80045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13572024-10-30T17:48:46.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.102357903s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13582024-10-30T17:48:46.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 655.020937ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13592024-10-30T17:48:46.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.269106353s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13602024-10-30T17:48:46.355ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 932.007415ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13612024-10-30T17:48:46.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.438115382s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13622024-10-30T17:48:46.409ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.003792175s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13632024-10-30T17:48:46.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.243132315s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13642024-10-30T17:48:46.780ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 695.81333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13652024-10-30T17:48:46.937ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.071627169s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13662024-10-30T17:48:47.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.205875386s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13672024-10-30T17:48:47.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 824.704463ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13682024-10-30T17:48:47.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.158143997s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13692024-10-30T17:48:47.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 1.040701839s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13702024-10-30T17:48:47.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 516.398177ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13712024-10-30T17:48:47.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 800.757344ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13722024-10-30T17:48:47.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.365771105s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13732024-10-30T17:48:47.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 880.21229ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13742024-10-30T17:48:47.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 915.411023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13752024-10-30T17:48:47.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 696.360874ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13762024-10-30T17:48:47.876ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.152840935s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13772024-10-30T17:48:47.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 768.506971ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13782024-10-30T17:48:48.021ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 955.10097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13792024-10-30T17:48:48.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 999.193763ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13802024-10-30T17:48:48.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 731.143966ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13812024-10-30T17:48:48.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 553.3737ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13822024-10-30T17:48:48.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 923.801749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13832024-10-30T17:48:48.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.01892612s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13842024-10-30T17:48:48.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 809.343443ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13852024-10-30T17:48:48.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.282373352s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
13862024-10-30T17:48:48.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.119958544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
13872024-10-30T17:48:48.863ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.358694767s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
13882024-10-30T17:48:48.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 808.541551ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13892024-10-30T17:48:48.984ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 537.631103ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13902024-10-30T17:48:48.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 518.706491ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13912024-10-30T17:48:49.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.069207322s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
13922024-10-30T17:48:49.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.075742779s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
13932024-10-30T17:48:49.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.133908939s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
13942024-10-30T17:48:49.350ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.479307214s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
13952024-10-30T17:48:49.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 707.228315ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
13962024-10-30T17:48:49.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.226692718s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
13972024-10-30T17:48:49.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.473597867s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
13982024-10-30T17:48:49.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 751.909596ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
13992024-10-30T17:48:49.849ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14002024-10-30T17:48:49.849ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14012024-10-30T17:48:49.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 751.792236ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
14022024-10-30T17:48:49.997ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 759.611502ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
14032024-10-30T17:48:50.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 68.728242ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14042024-10-30T17:48:50.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 633.838055ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
14052024-10-30T17:48:50.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 56.492553ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14062024-10-30T17:48:50.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 795.982594ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
14072024-10-30T17:48:50.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 641.184541ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
14082024-10-30T17:48:50.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 296.342642ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14092024-10-30T17:48:50.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 889.14919ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
14102024-10-30T17:48:50.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.347023927s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
14112024-10-30T17:48:50.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 1.371168438s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
14122024-10-30T17:48:50.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 233.458066ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14132024-10-30T17:48:50.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.16646687s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
14142024-10-30T17:48:50.640ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14152024-10-30T17:48:50.640ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14162024-10-30T17:48:50.646ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_7c337fba-b737-4f90-a327-065b4f3b4476/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='470e2a1a-48e6-4b7c-85f4-ff174820dd69'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14172024-10-30T17:48:50.646ZINFOSledAgent (ServiceManager): Booting oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14182024-10-30T17:48:50.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 522.116008ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
14192024-10-30T17:48:50.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.334620549s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
14202024-10-30T17:48:50.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.073553208s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
14212024-10-30T17:48:50.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 958.621892ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14222024-10-30T17:48:50.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 626.523759ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
14232024-10-30T17:48:50.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 46.058778ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14242024-10-30T17:48:50.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 756.740945ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
14252024-10-30T17:48:50.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 51.035937ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14262024-10-30T17:48:50.938ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 101.317513ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14272024-10-30T17:48:50.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 65.911595ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14282024-10-30T17:48:51.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.396505104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
14292024-10-30T17:48:51.006ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14302024-10-30T17:48:51.006ZINFOSledAgent (ServiceManager): Booting oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14312024-10-30T17:48:51.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 283.132891ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14322024-10-30T17:48:51.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.07256697s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
14332024-10-30T17:48:51.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 162.778881ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14342024-10-30T17:48:51.207ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 48.875962ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14352024-10-30T17:48:51.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 471.434632ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14362024-10-30T17:48:51.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 127.608934ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14372024-10-30T17:48:51.281ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.148045602s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
14382024-10-30T17:48:51.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.469717195s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
14392024-10-30T17:48:51.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 442.692219ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14402024-10-30T17:48:51.415ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 207.695277ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14412024-10-30T17:48:51.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 588.361888ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
14422024-10-30T17:48:51.648ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 599.743278ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14432024-10-30T17:48:51.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 545.109065ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
14442024-10-30T17:48:51.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 427.53594ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14452024-10-30T17:48:51.771ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 997.208805ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14462024-10-30T17:48:51.806ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 589.69307ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
14472024-10-30T17:48:51.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.088076244s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14482024-10-30T17:48:51.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.249488958s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
14492024-10-30T17:48:51.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 814.984987ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
14502024-10-30T17:48:51.882ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 1.32761261s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
14512024-10-30T17:48:51.943ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14522024-10-30T17:48:51.943ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14532024-10-30T17:48:52.087ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 897.694269ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
14542024-10-30T17:48:52.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.214398036s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
14552024-10-30T17:48:52.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 972.400708ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
14562024-10-30T17:48:52.180ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 697.53779ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14572024-10-30T17:48:52.226ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.097111517s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
14582024-10-30T17:48:52.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 911.092083ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14592024-10-30T17:48:52.295ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 71.15104ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14602024-10-30T17:48:52.322ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14612024-10-30T17:48:52.322ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
14622024-10-30T17:48:52.377ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 145.91262ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14632024-10-30T17:48:52.406ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 732.620225ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
14642024-10-30T17:48:52.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.115594472s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
14652024-10-30T17:48:52.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.337291294s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
14662024-10-30T17:48:52.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 251.041118ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14672024-10-30T17:48:52.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.072461063s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
14682024-10-30T17:48:52.701ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14692024-10-30T17:48:52.701ZINFOSledAgent (ServiceManager): Booting oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
14702024-10-30T17:48:52.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.027909812s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14712024-10-30T17:48:52.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.381421251s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
14722024-10-30T17:48:52.794ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 370.139467ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14732024-10-30T17:48:52.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 35.801936ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
14742024-10-30T17:48:52.886ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 1.152379909s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
14752024-10-30T17:48:52.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 135.603718ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
14762024-10-30T17:48:52.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.122714661s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
14772024-10-30T17:48:52.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 613.333637ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
14782024-10-30T17:48:53.045ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 102.583271ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
14792024-10-30T17:48:53.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 764.469547ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
14802024-10-30T17:48:53.120ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 594.571484ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
14812024-10-30T17:48:53.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 607.719741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
14822024-10-30T17:48:53.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 252.77942ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
14832024-10-30T17:48:53.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 561.395903ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14842024-10-30T17:48:53.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.193847472s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
14852024-10-30T17:48:53.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 831.283671ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
14862024-10-30T17:48:53.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 836.883845ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
14872024-10-30T17:48:53.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.072798937s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
14882024-10-30T17:48:53.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 926.070492ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
14892024-10-30T17:48:53.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 979.892021ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
14902024-10-30T17:48:53.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 956.32801ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
14912024-10-30T17:48:53.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 842.660535ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
14922024-10-30T17:48:53.744ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 784.536395ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
14932024-10-30T17:48:53.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.128261796s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
14942024-10-30T17:48:53.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 653.383238ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
14952024-10-30T17:48:53.790ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 728.926658ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
14962024-10-30T17:48:53.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.031225824s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
14972024-10-30T17:48:54.055ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 879.91401ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
14982024-10-30T17:48:54.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 986.768676ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
14992024-10-30T17:48:54.156ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 724.357209ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
15002024-10-30T17:48:54.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 614.972987ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
15012024-10-30T17:48:54.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 839.996781ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
15022024-10-30T17:48:54.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.104979945s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
15032024-10-30T17:48:54.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 1.105803968s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
15042024-10-30T17:48:54.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.14643593s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
15052024-10-30T17:48:54.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 559.436403ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
15062024-10-30T17:48:54.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 617.789709ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
15072024-10-30T17:48:54.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.232417108s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
15082024-10-30T17:48:54.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.230627602s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
15092024-10-30T17:48:54.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.300862304s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
15102024-10-30T17:48:54.578ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 664.172855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
15112024-10-30T17:48:54.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.43003104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
15122024-10-30T17:48:54.826ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.254722658s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
15132024-10-30T17:48:54.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 1.46651445s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
15142024-10-30T17:48:54.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 566.213734ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
15152024-10-30T17:48:54.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.039393123s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
15162024-10-30T17:48:55.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.485654232s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
15172024-10-30T17:48:55.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.339672376s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
15182024-10-30T17:48:55.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.068656531s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
15192024-10-30T17:48:55.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 676.960065ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
15202024-10-30T17:48:55.144ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.360080009s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
15212024-10-30T17:48:55.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 687.436567ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
15222024-10-30T17:48:55.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 990.974117ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
15232024-10-30T17:48:55.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.013338232s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
15242024-10-30T17:48:55.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 693.263899ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
15252024-10-30T17:48:55.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.29793184s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
15262024-10-30T17:48:55.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.495074584s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
15272024-10-30T17:48:55.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.393399561s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
15282024-10-30T17:48:55.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 631.096852ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
15292024-10-30T17:48:55.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.372855381s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
15302024-10-30T17:48:55.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 807.198288ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
15312024-10-30T17:48:55.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.454634948s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
15322024-10-30T17:48:56.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 686.996571ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
15332024-10-30T17:48:56.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.360230654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
15342024-10-30T17:48:56.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 964.98008ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
15352024-10-30T17:48:56.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 1.003467456s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
15362024-10-30T17:48:56.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 1.341790337s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
15372024-10-30T17:48:56.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.326428358s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
15382024-10-30T17:48:56.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 865.914121ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
15392024-10-30T17:48:56.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 980.538593ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
15402024-10-30T17:48:56.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.453344707s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
15412024-10-30T17:48:56.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.47727905s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
15422024-10-30T17:48:56.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.417753609s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
15432024-10-30T17:48:56.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.39439715s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
15442024-10-30T17:48:56.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 668.720884ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
15452024-10-30T17:48:56.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.071347151s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
15462024-10-30T17:48:57.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 990.447831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
15472024-10-30T17:48:57.188ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 564.60098ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
15482024-10-30T17:48:57.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 525.899519ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
15492024-10-30T17:48:57.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 749.258703ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
15502024-10-30T17:48:57.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 695.633785ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
15512024-10-30T17:48:57.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.181572834s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
15522024-10-30T17:48:57.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 751.070891ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
15532024-10-30T17:48:57.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.154032675s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
15542024-10-30T17:48:57.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 631.936378ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
15552024-10-30T17:48:57.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.479242126s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
15562024-10-30T17:48:57.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 514.522137ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
15572024-10-30T17:48:57.722ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 850.231109ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
15582024-10-30T17:48:57.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 834.838622ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
15592024-10-30T17:48:57.766ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 976.717817ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
15602024-10-30T17:48:57.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 721.004102ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
15612024-10-30T17:48:57.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 526.024085ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
15622024-10-30T17:48:57.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.021721744s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
15632024-10-30T17:48:57.983ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.099318217s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
15642024-10-30T17:48:57.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.29192131s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
15652024-10-30T17:48:58.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 942.967687ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
15662024-10-30T17:48:58.101ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 965.87996ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
15672024-10-30T17:48:58.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 826.898739ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
15682024-10-30T17:48:58.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 617.274048ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
15692024-10-30T17:48:58.202ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 791.388609ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
15702024-10-30T17:48:58.218ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 41.567247ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
15712024-10-30T17:48:58.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.273876758s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
15722024-10-30T17:48:58.269ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 123.27707ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
15732024-10-30T17:48:58.405ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 111.548251ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
15742024-10-30T17:48:58.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 648.105546ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
15752024-10-30T17:48:58.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.411071668s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
15762024-10-30T17:48:58.526ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 251.803314ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
15772024-10-30T17:48:58.583ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 1.467464614s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
15782024-10-30T17:48:58.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 988.294621ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
15792024-10-30T17:48:58.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 753.870229ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
15802024-10-30T17:48:58.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.031934931s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
15812024-10-30T17:48:58.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.48118244s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
15822024-10-30T17:48:58.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.452943639s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
15832024-10-30T17:48:58.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 425.484062ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
15842024-10-30T17:48:58.941ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15852024-10-30T17:48:58.941ZINFOSledAgent (ServiceManager): Booting oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
15862024-10-30T17:48:58.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.036638672s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
15872024-10-30T17:48:58.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 876.470744ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
15882024-10-30T17:48:58.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 559.936301ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
15892024-10-30T17:48:59.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.405891288s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
15902024-10-30T17:48:59.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 734.143523ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
15912024-10-30T17:48:59.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 980.05926ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
15922024-10-30T17:48:59.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.406764992s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
15932024-10-30T17:48:59.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 805.879252ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
15942024-10-30T17:48:59.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 36.131377ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
15952024-10-30T17:48:59.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 91.601779ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
15962024-10-30T17:48:59.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 656.040476ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
15972024-10-30T17:48:59.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.362034779s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
15982024-10-30T17:48:59.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 169.948488ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
15992024-10-30T17:48:59.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.364491325s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16002024-10-30T17:48:59.527ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.097643925s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
16012024-10-30T17:48:59.542ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 376.971414ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
16022024-10-30T17:48:59.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 965.309619ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
16032024-10-30T17:48:59.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 936.32429ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
16042024-10-30T17:48:59.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.130517027s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
16052024-10-30T17:48:59.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 998.737243ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
16062024-10-30T17:48:59.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 542.896606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
16072024-10-30T17:48:59.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 1.497886426s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
16082024-10-30T17:48:59.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.152012929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
16092024-10-30T17:48:59.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 772.229731ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
16102024-10-30T17:48:59.979ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 902.001088ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
16112024-10-30T17:49:00.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.005859417s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
16122024-10-30T17:49:00.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 1.336583491s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
16132024-10-30T17:49:00.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 629.831995ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
16142024-10-30T17:49:00.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.058910919s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
16152024-10-30T17:49:00.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 579.230841ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
16162024-10-30T17:49:00.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.482168119s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
16172024-10-30T17:49:00.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.40605573s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
16182024-10-30T17:49:00.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.093553155s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
16192024-10-30T17:49:00.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 550.633642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
16202024-10-30T17:49:00.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 1.144813556s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
16212024-10-30T17:49:00.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.317148242s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
16222024-10-30T17:49:00.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 743.854857ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
16232024-10-30T17:49:00.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.470174176s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
16242024-10-30T17:49:00.761ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 1.2143271s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
16252024-10-30T17:49:00.788ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.061085564s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16262024-10-30T17:49:00.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 1.152836493s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
16272024-10-30T17:49:00.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.236622273s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
16282024-10-30T17:49:00.877ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.376584697s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
16292024-10-30T17:49:00.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 670.412178ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
16302024-10-30T17:49:01.042ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.422648609s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
16312024-10-30T17:49:01.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.063850917s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
16322024-10-30T17:49:01.114ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 989.025029ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
16332024-10-30T17:49:01.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.343281842s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
16342024-10-30T17:49:01.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 1.25491295s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
16352024-10-30T17:49:01.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 1.123312609s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
16362024-10-30T17:49:01.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 933.835861ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
16372024-10-30T17:49:01.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.184686723s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
16382024-10-30T17:49:01.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.086176928s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
16392024-10-30T17:49:01.719ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 1.149975085s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
16402024-10-30T17:49:01.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 628.438147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16412024-10-30T17:49:01.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.107550978s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
16422024-10-30T17:49:01.916ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.011127403s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
16432024-10-30T17:49:01.970ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 976.157915ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
16442024-10-30T17:49:01.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 1.082077798s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
16452024-10-30T17:49:01.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 1.308143983s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
16462024-10-30T17:49:02.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 577.072517ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
16472024-10-30T17:49:02.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.126066386s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
16482024-10-30T17:49:02.178ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.0199177s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
16492024-10-30T17:49:02.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.058739356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
16502024-10-30T17:49:02.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 541.56125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
16512024-10-30T17:49:02.294ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 5.709454324s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
16522024-10-30T17:49:02.408ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.283070715s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
16532024-10-30T17:49:02.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 933.759021ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
16542024-10-30T17:49:02.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 754.957353ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16552024-10-30T17:49:02.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 1.034456642s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
16562024-10-30T17:49:02.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 1.420305777s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
16572024-10-30T17:49:02.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 779.36642ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
16582024-10-30T17:49:02.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 983.762402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
16592024-10-30T17:49:02.729ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 504.330277ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
16602024-10-30T17:49:02.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.350205303s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
16612024-10-30T17:49:02.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 684.409552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
16622024-10-30T17:49:02.879ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 537.131302ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
16632024-10-30T17:49:02.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.192307053s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
16642024-10-30T17:49:02.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82") failed: Property not found. retry in 1.319283278s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
16652024-10-30T17:49:02.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.012492492s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
16662024-10-30T17:49:03.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c") failed: Property not found. retry in 1.142217096s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
16672024-10-30T17:49:03.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 963.623992ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
16682024-10-30T17:49:03.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 869.070813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
16692024-10-30T17:49:03.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 1.196916546s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
16702024-10-30T17:49:03.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 782.483666ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16712024-10-30T17:49:03.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127") failed: Property not found. retry in 1.277466618s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
16722024-10-30T17:49:03.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 1.318163995s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
16732024-10-30T17:49:03.427ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.148198528s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
16742024-10-30T17:49:03.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 1.229305167s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
16752024-10-30T17:49:03.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 938.366011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
16762024-10-30T17:49:03.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 544.791813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
16772024-10-30T17:49:03.617ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 1.249051479s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
16782024-10-30T17:49:03.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 847.242517ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
16792024-10-30T17:49:03.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 716.254415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
16802024-10-30T17:49:04.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 749.861868ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
16812024-10-30T17:49:04.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 821.652275ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16822024-10-30T17:49:04.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36") failed: Property not found. retry in 1.075401927s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
16832024-10-30T17:49:04.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d") failed: Property not found. retry in 930.549104ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
16842024-10-30T17:49:04.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.014776912s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
16852024-10-30T17:49:04.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 866.21682ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
16862024-10-30T17:49:04.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 548.831231ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
16872024-10-30T17:49:04.195ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 1.436199684s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
16882024-10-30T17:49:04.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c") failed: Property not found. retry in 765.359039ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
16892024-10-30T17:49:04.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146") failed: Property not found. retry in 1.236705789s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
16902024-10-30T17:49:04.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3") failed: Property not found. retry in 981.902668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
16912024-10-30T17:49:04.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 869.241817ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
16922024-10-30T17:49:04.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb") failed: Property not found. retry in 1.167581743s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
16932024-10-30T17:49:04.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 1.38403766s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
16942024-10-30T17:49:04.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 744.058749ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
16952024-10-30T17:49:04.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 987.253963ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
16962024-10-30T17:49:04.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245") failed: Property not found. retry in 1.298352408s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
16972024-10-30T17:49:04.893ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 524.026535ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
16982024-10-30T17:49:04.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90") failed: Property not found. retry in 1.195721982s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
16992024-10-30T17:49:05.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d") failed: Property not found. retry in 1.223295163s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
17002024-10-30T17:49:05.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 520.696412ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
17012024-10-30T17:49:05.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 1.494950525s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
17022024-10-30T17:49:05.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574") failed: Property not found. retry in 1.357589032s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
17032024-10-30T17:49:05.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8") failed: Property not found. retry in 1.273047852s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
17042024-10-30T17:49:05.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d") failed: Property not found. retry in 922.474038ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
17052024-10-30T17:49:05.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346") failed: Property not found. retry in 1.143143761s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
17062024-10-30T17:49:05.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910") failed: Property not found. retry in 1.380089857s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
17072024-10-30T17:49:06.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 1.14745514s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
17082024-10-30T17:49:06.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703") failed: Property not found. retry in 1.363945535s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
17092024-10-30T17:49:07.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 1.303344691s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
17102024-10-30T17:49:08.057ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 8.463345276s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
17112024-10-30T17:49:08.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5") failed: Property not found. retry in 723.21224ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
17122024-10-30T17:49:08.660ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48650
17132024-10-30T17:49:08.661ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 73 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48650 req_id = a5374765-9292-46dc-92c5-ea71036b1484 response_code = 200 uri = /baseboard
17142024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_clickhouse_366b95de-05cf-43cb-8b58-0d4b6e1b2574
17152024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_pantry_eaea67ce-5679-46bf-b0bf-4a021e5bbec3
17162024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_fb3cfc2e-5d66-4180-9b7b-4b7fa14a9346
17172024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_530c539e-4272-4675-882c-e4d1d574c146
17182024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_external_dns_ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703
17192024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_oximeter_7f02d345-3366-4b62-9cd8-995f8742dd0c
17202024-10-30T17:49:09.292ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_0206d1cc-2f17-4c82-9c67-f4c41faa893d
17212024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_04ada316-c920-4e4a-9d41-4c467b02ec90
17222024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_nexus_c24ae96c-052c-4566-8772-4d7cffb20cb5
17232024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_a9aa814a-015b-4f38-8445-d17789da9b82
17242024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_external_dns_185f90b2-80bf-479a-b8b0-ec1f3c02a2fb
17252024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_pantry_ea535df2-fd38-4652-9e6f-02fe39690a9c
17262024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_7180a284-fbab-4f51-8bb9-97f62b818245
17272024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_nexus_084cb44c-d8f2-4ea3-8498-21306b882ab8
17282024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_0be13225-a78b-4173-9c0a-759ebd323d2d
17292024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_34b851f8-bee3-4493-8196-84205c8c2127
17302024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_nexus_ee372901-c3d9-45f0-8c4f-5c4702a28b4d
17312024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_pantry_f490660a-0116-4f0f-b1bc-faa5f011ce36
17322024-10-30T17:49:09.293ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3272 zone = oxz_crucible_c61c51d9-a3b2-4ccb-ac02-b78ef7496910
17332024-10-30T17:49:09.294ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17342024-10-30T17:49:09.296ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17352024-10-30T17:49:09.298ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 39344888 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:59677 req_id = d594beb8-2bc7-44cd-9a0a-4512cfbd13aa response_code = 204 uri = /omicron-zones
17362024-10-30T17:49:09.302ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1321
17372024-10-30T17:49:09.303ZINFOSledAgent (RSS): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
17382024-10-30T17:49:09.303ZINFOSledAgent (RSS): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
17392024-10-30T17:49:09.306ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
17402024-10-30T17:49:09.307ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::a]:12221 file = sled-agent/src/rack_setup/service.rs:719
17412024-10-30T17:49:09.338ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, rib_priority: None, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:886
17422024-10-30T17:49:09.339ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17432024-10-30T17:49:09.462ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17442024-10-30T17:49:09.624ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17452024-10-30T17:49:09.762ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17462024-10-30T17:49:10.021ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17472024-10-30T17:49:10.333ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17482024-10-30T17:49:10.693ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17492024-10-30T17:49:11.074ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17502024-10-30T17:49:11.477ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17512024-10-30T17:49:11.785ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17522024-10-30T17:49:12.474ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17532024-10-30T17:49:12.991ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17542024-10-30T17:49:14.027ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17552024-10-30T17:49:15.277ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17562024-10-30T17:49:16.556ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 11.348032223s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
17572024-10-30T17:49:16.771ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/5f1b3995-6708-40e4-9d28-234b43b9cddc/initialization-complete) file = sled-agent/src/rack_setup/service.rs:948
17582024-10-30T17:49:18.074ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "1ca3fba4-43ed-46cd-8f79-5f2304d55f22", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:18 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"5f1b3995-6708-40e4-9d28-234b43b9cddc\"", request_id: "1ca3fba4-43ed-46cd-8f79-5f2304d55f22" } file = sled-agent/src/rack_setup/service.rs:948
17592024-10-30T17:49:19.191ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "6662ab0d-758a-4476-a3a8-2237bb851dbf", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:19 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"5f1b3995-6708-40e4-9d28-234b43b9cddc\"", request_id: "6662ab0d-758a-4476-a3a8-2237bb851dbf" } file = sled-agent/src/rack_setup/service.rs:948
17602024-10-30T17:49:21.577ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "1f9699a7-d166-4918-b773-64d1dec76c3e", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:21 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "1f9699a7-d166-4918-b773-64d1dec76c3e" } file = sled-agent/src/rack_setup/service.rs:948
17612024-10-30T17:49:23.597ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "21dcaf62-8ccc-417b-9715-2f7c4ef62a08", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:23 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "21dcaf62-8ccc-417b-9715-2f7c4ef62a08" } file = sled-agent/src/rack_setup/service.rs:948
17622024-10-30T17:49:26.888ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "039d7433-55fe-46fe-b6cc-80052110f1d3", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:26 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "039d7433-55fe-46fe-b6cc-80052110f1d3" } file = sled-agent/src/rack_setup/service.rs:948
17632024-10-30T17:49:30.123ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "447df30b-233a-4dbe-bcfc-9b21dda34401", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:30 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "447df30b-233a-4dbe-bcfc-9b21dda34401" } file = sled-agent/src/rack_setup/service.rs:948
17642024-10-30T17:49:33.952ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "7b470c78-c87d-4fe1-8507-c4d4e3d5ed54", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:33 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "7b470c78-c87d-4fe1-8507-c4d4e3d5ed54" } file = sled-agent/src/rack_setup/service.rs:948
17652024-10-30T17:49:37.129ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:56704 remote_addr = [fd00:1122:3344:101::d]:63935 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
17662024-10-30T17:49:37.143ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 14086 local_addr = [fd00:1122:3344:101::1]:56704 method = GET remote_addr = [fd00:1122:3344:101::d]:63935 req_id = 046cfa84-8361-40eb-a866-c2388d1af382 response_code = 200 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 uri = /297da6c0-3dc5-4f20-9a57-48a339550273
17672024-10-30T17:49:37.862ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "29e844b2-5812-4318-9ec7-1c343e9967ab", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:37 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "29e844b2-5812-4318-9ec7-1c343e9967ab" } file = sled-agent/src/rack_setup/service.rs:948
17682024-10-30T17:49:38.693ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:54910
17692024-10-30T17:49:38.693ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:54910 req_id = b1964e8f-8912-41a1-897a-856768680acf response_code = 200 uri = /baseboard
17702024-10-30T17:49:41.142ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "2e52058d-220f-4f7e-912f-51a3db93ded9", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:41 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "2e52058d-220f-4f7e-912f-51a3db93ded9" } file = sled-agent/src/rack_setup/service.rs:948
17712024-10-30T17:49:51.772ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "360c6606-a761-4915-8a20-b01f858c23b2", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:50 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "360c6606-a761-4915-8a20-b01f858c23b2" } file = sled-agent/src/rack_setup/service.rs:948
17722024-10-30T17:49:57.011ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "bb01859d-8415-4182-bd1c-707a99618500", "content-length": "173", "date": "Wed, 30 Oct 2024 17:49:57 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"297da6c0-3dc5-4f20-9a57-48a339550273\"", request_id: "bb01859d-8415-4182-bd1c-707a99618500" } file = sled-agent/src/rack_setup/service.rs:948
17732024-10-30T17:50:02.064ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:500 new_generation = 1 old_generation = 1 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
17742024-10-30T17:50:02.148ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50899
17752024-10-30T17:50:02.149ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17762024-10-30T17:50:02.149ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17772024-10-30T17:50:02.150ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
17782024-10-30T17:50:02.151ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
17792024-10-30T17:50:02.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 208129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50899 req_id = 260883e0-d22a-492e-84f9-5b271af17d59 response_code = 200 uri = /inventory
17802024-10-30T17:50:04.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43585
17812024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0817b15f-d0fb-4e3e-8619-51d448476a89, kind: Service { id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 }, name: Name("external-dns-ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 59])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 25610ba7-70b1-4917-a427-f5070892a54f, kind: Service { id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb }, name: Name("external-dns-185f90b2-80bf-479a-b8b0-ec1f3c02a2fb"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 105])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6734898c-60d3-45c1-8b48-7163e5e7e5d7, kind: Service { id: ee372901-c3d9-45f0-8c4f-5c4702a28b4d }, name: Name("nexus-ee372901-c3d9-45f0-8c4f-5c4702a28b4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 33])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a90bbb13-856b-46a6-8e82-4b9742f6c506, kind: Service { id: c24ae96c-052c-4566-8772-4d7cffb20cb5 }, name: Name("nexus-c24ae96c-052c-4566-8772-4d7cffb20cb5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6a0dd9f-e087-4f2a-b710-699e007ce774, kind: Service { id: 084cb44c-d8f2-4ea3-8498-21306b882ab8 }, name: Name("nexus-084cb44c-d8f2-4ea3-8498-21306b882ab8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17822024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17832024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17842024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
17852024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17862024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17872024-10-30T17:50:04.177ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17882024-10-30T17:50:04.177ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1818 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43585 req_id = d0ff6d1d-1d93-4a84-bbb6-daa38cf9e859 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
17892024-10-30T17:50:04.178ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:958
17902024-10-30T17:50:04.517ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41066
17912024-10-30T17:50:04.517ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17922024-10-30T17:50:04.517ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17932024-10-30T17:50:04.517ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/config/omicron-zones.json file = common/src/ledger.rs:177
17942024-10-30T17:50:04.518ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/config/omicron-zones.json file = common/src/ledger.rs:177
17952024-10-30T17:50:04.739ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 221942 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41066 req_id = 568a2a39-6cb9-4145-b859-f26a05edae2d response_code = 200 uri = /inventory
17962024-10-30T17:50:04.807ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51719
17972024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0817b15f-d0fb-4e3e-8619-51d448476a89, kind: Service { id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 }, name: Name("external-dns-ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 59])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 25610ba7-70b1-4917-a427-f5070892a54f, kind: Service { id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb }, name: Name("external-dns-185f90b2-80bf-479a-b8b0-ec1f3c02a2fb"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 105])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6734898c-60d3-45c1-8b48-7163e5e7e5d7, kind: Service { id: ee372901-c3d9-45f0-8c4f-5c4702a28b4d }, name: Name("nexus-ee372901-c3d9-45f0-8c4f-5c4702a28b4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 33])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a90bbb13-856b-46a6-8e82-4b9742f6c506, kind: Service { id: c24ae96c-052c-4566-8772-4d7cffb20cb5 }, name: Name("nexus-c24ae96c-052c-4566-8772-4d7cffb20cb5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6a0dd9f-e087-4f2a-b710-699e007ce774, kind: Service { id: 084cb44c-d8f2-4ea3-8498-21306b882ab8 }, name: Name("nexus-084cb44c-d8f2-4ea3-8498-21306b882ab8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17982024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17992024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18002024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
18012024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18022024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18032024-10-30T17:50:04.807ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18042024-10-30T17:50:04.807ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 320 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51719 req_id = b0c709a6-4859-40bf-9e14-d1f3fd98fe72 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18052024-10-30T17:50:05.172ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56487
18062024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0817b15f-d0fb-4e3e-8619-51d448476a89, kind: Service { id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 }, name: Name("external-dns-ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 59])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 25610ba7-70b1-4917-a427-f5070892a54f, kind: Service { id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb }, name: Name("external-dns-185f90b2-80bf-479a-b8b0-ec1f3c02a2fb"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 105])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6734898c-60d3-45c1-8b48-7163e5e7e5d7, kind: Service { id: ee372901-c3d9-45f0-8c4f-5c4702a28b4d }, name: Name("nexus-ee372901-c3d9-45f0-8c4f-5c4702a28b4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 33])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a90bbb13-856b-46a6-8e82-4b9742f6c506, kind: Service { id: c24ae96c-052c-4566-8772-4d7cffb20cb5 }, name: Name("nexus-c24ae96c-052c-4566-8772-4d7cffb20cb5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6a0dd9f-e087-4f2a-b710-699e007ce774, kind: Service { id: 084cb44c-d8f2-4ea3-8498-21306b882ab8 }, name: Name("nexus-084cb44c-d8f2-4ea3-8498-21306b882ab8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18072024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18082024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18092024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
18102024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18112024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18122024-10-30T17:50:05.172ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18132024-10-30T17:50:05.172ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56487 req_id = ee4782ac-d799-463f-a07d-a25cb7f584dd response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18142024-10-30T17:50:05.403ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34857
18152024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:731 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 0817b15f-d0fb-4e3e-8619-51d448476a89, kind: Service { id: ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703 }, name: Name("external-dns-ed6e3d61-a5e7-4311-bac8-f8f1e5e8a703"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 230, 59])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 25610ba7-70b1-4917-a427-f5070892a54f, kind: Service { id: 185f90b2-80bf-479a-b8b0-ec1f3c02a2fb }, name: Name("external-dns-185f90b2-80bf-479a-b8b0-ec1f3c02a2fb"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 239, 105])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 6734898c-60d3-45c1-8b48-7163e5e7e5d7, kind: Service { id: ee372901-c3d9-45f0-8c4f-5c4702a28b4d }, name: Name("nexus-ee372901-c3d9-45f0-8c4f-5c4702a28b4d"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 248, 33])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: a90bbb13-856b-46a6-8e82-4b9742f6c506, kind: Service { id: c24ae96c-052c-4566-8772-4d7cffb20cb5 }, name: Name("nexus-c24ae96c-052c-4566-8772-4d7cffb20cb5"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 170, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: e6a0dd9f-e087-4f2a-b710-699e007ce774, kind: Service { id: 084cb44c-d8f2-4ea3-8498-21306b882ab8 }, name: Name("nexus-084cb44c-d8f2-4ea3-8498-21306b882ab8"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 130, 23])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18162024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18172024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18182024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte0" rules = []
18192024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18202024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18212024-10-30T17:50:05.403ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:750 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18222024-10-30T17:50:05.403ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 372 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34857 req_id = 57a2c429-a823-4506-93e8-d79dadebbc63 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18232024-10-30T17:50:08.728ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53641
18242024-10-30T17:50:08.728ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 32 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53641 req_id = 954b859a-674c-40c1-b959-6e5ed3783c38 response_code = 200 uri = /baseboard
18252024-10-30T17:50:10.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64600
18262024-10-30T17:50:10.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64600 req_id = 152956a7-6dd2-4d2b-8888-090d361084f6 response_code = 200 uri = /vpc-routes
18272024-10-30T17:50:10.886ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18282024-10-30T17:50:10.886ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 265 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64600 req_id = ec9a1744-76d6-46a7-b84a-5c51f46cc5e8 response_code = 204 uri = /eip-gateways
18292024-10-30T17:50:10.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2019 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:64600 req_id = 3fb3264e-3e11-4070-9212-ef7ca440c16c response_code = 204 uri = /vpc-routes
18302024-10-30T17:50:19.801ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47546
18312024-10-30T17:50:19.803ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47546 req_id = e2bb2c2b-b3a2-40a6-bdc8-61a555c97dca response_code = 200 uri = /vpc-routes
18322024-10-30T17:50:19.866ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18332024-10-30T17:50:19.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47546 req_id = 2dc3b36f-a637-47b9-8870-6cbf9b41dd11 response_code = 204 uri = /eip-gateways
18342024-10-30T17:50:19.878ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 281 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:47546 req_id = ec6d74fb-a875-463c-b97f-32499f06ce41 response_code = 204 uri = /vpc-routes
18352024-10-30T17:50:20.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51268
18362024-10-30T17:50:20.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51268 req_id = a7a54e5f-251e-4565-9d8f-8cad3f4edc5f response_code = 200 uri = /v2p
18372024-10-30T17:50:20.034ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62296
18382024-10-30T17:50:20.034ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 29 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62296 req_id = 348e4e12-f4a0-4a64-9559-2452f0a25e9e response_code = 200 uri = /vpc-routes
18392024-10-30T17:50:20.082ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62662
18402024-10-30T17:50:20.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62662 req_id = 35aa070b-44e5-419e-b082-0d79d4cabad9 response_code = 200 uri = /vpc-routes
18412024-10-30T17:50:20.106ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18422024-10-30T17:50:20.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62296 req_id = 441a5873-1cb8-4ce6-812b-70673e06998a response_code = 204 uri = /eip-gateways
18432024-10-30T17:50:20.160ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18442024-10-30T17:50:20.160ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62662 req_id = 25aa2da5-0e31-4410-b581-9bfaf0bd8534 response_code = 204 uri = /eip-gateways
18452024-10-30T17:50:20.210ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38631
18462024-10-30T17:50:20.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38631 req_id = c37ed8e3-6c31-4562-809d-dd497045a048 response_code = 200 uri = /v2p
18472024-10-30T17:50:20.215ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:62662 req_id = 08df7d3c-d88c-44d6-b4ab-6721c0245c40 response_code = 204 uri = /vpc-routes
18482024-10-30T17:50:20.231ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61467
18492024-10-30T17:50:20.232ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61467 req_id = 1feb90ef-4b81-487f-b90e-9205cabb872a response_code = 200 uri = /v2p
18502024-10-30T17:50:20.239ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62296 req_id = 975463e7-a9b7-4033-8933-d1664ce6d613 response_code = 204 uri = /vpc-routes
18512024-10-30T17:50:20.343ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50977
18522024-10-30T17:50:20.343ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
18532024-10-30T17:50:20.404ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
18542024-10-30T17:50:20.457ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114501 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50977 req_id = 1bfa8f51-bc62-446a-9030-572bf4fcc923 response_code = 204 uri = /switch-ports
18552024-10-30T17:50:20.457ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49329
18562024-10-30T17:50:20.458ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35321
18572024-10-30T17:50:20.458ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
18582024-10-30T17:50:20.461ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3730 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50977 req_id = d46a03c3-31d2-4f94-af40-e6498e0163ab response_code = 200 uri = /network-bootstore-config
18592024-10-30T17:50:20.501ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
18602024-10-30T17:50:20.510ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
18612024-10-30T17:50:20.510ZINFOSledAgent (bootstore): Writing ledger to /pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18622024-10-30T17:50:20.510ZINFOSledAgent (bootstore): Writing ledger to /pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18632024-10-30T17:50:20.511ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
18642024-10-30T17:50:20.511ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2441 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:50977 req_id = fd6fa53e-8916-4fba-ad65-7af182de9440 response_code = 204 uri = /network-bootstore-config
18652024-10-30T17:50:20.554ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
18662024-10-30T17:50:20.554ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96076 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:49329 req_id = 257c8e8d-7a02-42be-bb5c-f03ae1d447a3 response_code = 204 uri = /switch-ports
18672024-10-30T17:50:20.559ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 791 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49329 req_id = 6ef09d01-2a9d-451e-b903-1ed58841e6a9 response_code = 200 uri = /network-bootstore-config
18682024-10-30T17:50:20.592ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
18692024-10-30T17:50:20.651ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 192752 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:35321 req_id = 377deb06-a3a8-4f18-9a5e-407c5b17562f response_code = 204 uri = /switch-ports
18702024-10-30T17:50:20.651ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35321 req_id = eebe0472-a75d-422c-a5c7-b1c0c64481a5 response_code = 200 uri = /network-bootstore-config
18712024-10-30T17:50:38.761ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:60470
18722024-10-30T17:50:38.761ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 37 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:60470 req_id = bc4fe965-01a6-4473-8bd3-32ac58c1da61 response_code = 200 uri = /baseboard
18732024-10-30T17:50:49.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49949
18742024-10-30T17:50:49.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49949 req_id = 6295014f-b487-439e-97f4-cd704df556a0 response_code = 200 uri = /vpc-routes
18752024-10-30T17:50:49.633ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18762024-10-30T17:50:49.633ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 75 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49949 req_id = 0d2d0678-759f-45ee-93ee-235bf8c7294e response_code = 204 uri = /eip-gateways
18772024-10-30T17:50:49.750ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61797
18782024-10-30T17:50:49.751ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61797 req_id = d747b4a2-c97f-46ee-b734-4ba994ed6b39 response_code = 200 uri = /v2p
18792024-10-30T17:50:49.756ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 200 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:49949 req_id = 9bbd29fa-42be-406b-b12b-5b383d421bd2 response_code = 204 uri = /vpc-routes
18802024-10-30T17:50:49.783ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42307
18812024-10-30T17:50:49.783ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42307 req_id = f1385f70-7104-427c-8f8d-d0d3bbb37e01 response_code = 200 uri = /vpc-routes
18822024-10-30T17:50:49.830ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53227
18832024-10-30T17:50:49.830ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 51 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53227 req_id = 8370e1a3-d532-4cfa-821e-9ae8b1c85823 response_code = 200 uri = /vpc-routes
18842024-10-30T17:50:49.834ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18852024-10-30T17:50:49.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 45 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42307 req_id = c90d958a-ef7d-41ba-88e3-a836e10169b6 response_code = 204 uri = /eip-gateways
18862024-10-30T17:50:49.844ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42307 req_id = 7777b980-363c-406d-8754-953247f417f7 response_code = 204 uri = /vpc-routes
18872024-10-30T17:50:49.934ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41874
18882024-10-30T17:50:49.934ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41874 req_id = 047a2824-aaaa-4598-bb97-7ec18af96213 response_code = 200 uri = /v2p
18892024-10-30T17:50:49.946ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52380
18902024-10-30T17:50:49.946ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
18912024-10-30T17:50:49.986ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
18922024-10-30T17:50:50.046ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99919 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:52380 req_id = 9d65e7e2-2d9b-4a4a-ad51-87d50db0dac1 response_code = 204 uri = /switch-ports
18932024-10-30T17:50:50.046ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56799
18942024-10-30T17:50:50.046ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60313
18952024-10-30T17:50:50.047ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
18962024-10-30T17:50:50.047ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
18972024-10-30T17:50:50.047ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 219 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53227 req_id = 529c4269-d012-4767-9d55-435eabaa2ca9 response_code = 204 uri = /eip-gateways
18982024-10-30T17:50:50.051ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4608 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56799 req_id = 6d12571b-cd4f-4739-a97a-06e7f4025677 response_code = 200 uri = /v2p
18992024-10-30T17:50:50.052ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4784 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52380 req_id = c804b5e8-bb9a-4a42-be06-f8884788f760 response_code = 200 uri = /network-bootstore-config
19002024-10-30T17:50:50.059ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 278 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53227 req_id = 9ff7749d-02ff-4c1e-9432-63035e5f49f7 response_code = 204 uri = /vpc-routes
19012024-10-30T17:50:50.088ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19022024-10-30T17:50:50.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94051 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60313 req_id = 56da7ab3-f576-4c10-9538-e90d95cafded response_code = 204 uri = /switch-ports
19032024-10-30T17:50:50.141ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60313 req_id = 5a6d54e4-d756-4d75-9382-e8947614ab76 response_code = 200 uri = /network-bootstore-config
19042024-10-30T17:50:50.165ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37945
19052024-10-30T17:50:50.166ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19062024-10-30T17:50:50.201ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19072024-10-30T17:50:50.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99106 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37945 req_id = 6dce9d6e-cc7b-44d1-80d1-16d793cfd19a response_code = 204 uri = /switch-ports
19082024-10-30T17:50:50.265ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37945 req_id = 08ea47e1-4ec3-4002-ba39-96754c822190 response_code = 200 uri = /network-bootstore-config
19092024-10-30T17:51:08.794ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58293
19102024-10-30T17:51:08.794ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 68 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58293 req_id = 65f934d3-8cb8-45c2-b239-43674b1a3930 response_code = 200 uri = /baseboard
19112024-10-30T17:51:19.838ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49687
19122024-10-30T17:51:19.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49687 req_id = ab7e060b-47bc-489f-ade9-c8f28dfdec99 response_code = 200 uri = /vpc-routes
19132024-10-30T17:51:19.841ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56213
19142024-10-30T17:51:19.841ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56213 req_id = 7246540a-ea12-4559-8f56-161857540596 response_code = 200 uri = /v2p
19152024-10-30T17:51:19.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59769
19162024-10-30T17:51:19.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59769 req_id = 6b1c1887-c5a4-4fcf-acbf-c439f9ab3cf8 response_code = 200 uri = /vpc-routes
19172024-10-30T17:51:19.869ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19182024-10-30T17:51:19.869ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59769 req_id = f18cb2bb-2aa8-4921-824b-35b5d7bd7c17 response_code = 204 uri = /eip-gateways
19192024-10-30T17:51:19.898ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19202024-10-30T17:51:19.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49687 req_id = 16005217-13b4-4ce1-bc76-60378f1833f8 response_code = 204 uri = /eip-gateways
19212024-10-30T17:51:20.030ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34291
19222024-10-30T17:51:20.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34291 req_id = f9a33682-3352-4ba6-9749-7a672fb17965 response_code = 200 uri = /vpc-routes
19232024-10-30T17:51:20.078ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19242024-10-30T17:51:20.078ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 43 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34291 req_id = 326aacf9-2cf5-43a6-b66e-75eb1af2f643 response_code = 204 uri = /eip-gateways
19252024-10-30T17:51:20.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59769 req_id = ddc876fd-e550-4717-ae87-ee421adee864 response_code = 204 uri = /vpc-routes
19262024-10-30T17:51:20.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 86 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:34291 req_id = b215f681-4c83-4d27-a603-82bed929cf01 response_code = 204 uri = /vpc-routes
19272024-10-30T17:51:20.128ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51374
19282024-10-30T17:51:20.128ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19292024-10-30T17:51:20.170ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19302024-10-30T17:51:20.223ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95383 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51374 req_id = 3aed0dd8-bb03-4604-a0a3-1144a72c401b response_code = 204 uri = /switch-ports
19312024-10-30T17:51:20.224ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50974
19322024-10-30T17:51:20.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50974 req_id = 2d1b5dcc-c535-4780-b341-dfaecc97c5c1 response_code = 200 uri = /v2p
19332024-10-30T17:51:20.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51374 req_id = 2cee15d8-3ace-4a47-814c-fa1afb09018c response_code = 200 uri = /network-bootstore-config
19342024-10-30T17:51:20.224ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37790
19352024-10-30T17:51:20.225ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37790 req_id = 716eaf9a-5a5e-4593-9d7c-80fad05b4630 response_code = 200 uri = /v2p
19362024-10-30T17:51:20.235ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49687 req_id = 34cbd7c7-8192-498e-a1e6-fdb54b07eb39 response_code = 204 uri = /vpc-routes
19372024-10-30T17:51:20.348ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61386
19382024-10-30T17:51:20.348ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19392024-10-30T17:51:20.385ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19402024-10-30T17:51:20.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92830 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:61386 req_id = 1011a554-5103-4670-b075-018dba0ffcb7 response_code = 204 uri = /switch-ports
19412024-10-30T17:51:20.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50362
19422024-10-30T17:51:20.441ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19432024-10-30T17:51:20.474ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19442024-10-30T17:51:20.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84477 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:50362 req_id = b3d0db5c-a44d-4bb6-8653-2cb7fef81f71 response_code = 204 uri = /switch-ports
19452024-10-30T17:51:20.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61386 req_id = cee7f6d2-ba0d-477b-a941-5c8dab486fb9 response_code = 200 uri = /network-bootstore-config
19462024-10-30T17:51:20.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50362 req_id = a63fe590-419f-44d2-8816-630355b60e6f response_code = 200 uri = /network-bootstore-config
19472024-10-30T17:51:38.828ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:62971
19482024-10-30T17:51:38.828ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 58 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:62971 req_id = 1ff154ae-34cd-404d-9cb0-0c6f141f8214 response_code = 200 uri = /baseboard
19492024-10-30T17:51:49.658ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65068
19502024-10-30T17:51:49.658ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65068 req_id = b9e2d708-ffc0-4721-84ed-3621d7b517ca response_code = 200 uri = /vpc-routes
19512024-10-30T17:51:49.729ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41843
19522024-10-30T17:51:49.729ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 183 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41843 req_id = e39e2474-79e2-45b6-9c18-7a8a03a6a568 response_code = 200 uri = /v2p
19532024-10-30T17:51:49.859ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57615
19542024-10-30T17:51:49.859ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19552024-10-30T17:51:49.913ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19562024-10-30T17:51:49.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137309 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57615 req_id = 9dda3754-023d-4d22-a91c-939148c540f9 response_code = 204 uri = /switch-ports
19572024-10-30T17:51:49.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52337
19582024-10-30T17:51:49.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38604
19592024-10-30T17:51:49.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51369
19602024-10-30T17:51:49.996ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19612024-10-30T17:51:49.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65068 req_id = 22ff3577-2199-4673-9bde-5479ef1d1be1 response_code = 204 uri = /eip-gateways
19622024-10-30T17:51:49.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 306 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57615 req_id = 842f0225-e41e-48a9-b2dc-69000ccfc8cc response_code = 200 uri = /network-bootstore-config
19632024-10-30T17:51:49.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51369 req_id = bc23d1a6-c935-4182-b221-abf5efa07b82 response_code = 200 uri = /vpc-routes
19642024-10-30T17:51:49.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 151 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38604 req_id = 9e1359c9-3aef-4667-bb17-0428349b982b response_code = 200 uri = /vpc-routes
19652024-10-30T17:51:49.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 377 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52337 req_id = a3abb44c-9583-497b-8816-a0aedb5c5c2e response_code = 200 uri = /v2p
19662024-10-30T17:51:50.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 445 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65068 req_id = bd8a6e4d-95e2-4370-8dbb-53451e13f4f9 response_code = 204 uri = /vpc-routes
19672024-10-30T17:51:50.023ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19682024-10-30T17:51:50.023ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51369 req_id = b8bf42e6-7c80-4e06-bcfb-a14ef682d40d response_code = 204 uri = /eip-gateways
19692024-10-30T17:51:50.053ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19702024-10-30T17:51:50.053ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38604 req_id = d246d98e-3764-4b66-acb4-28d4c4dca124 response_code = 204 uri = /eip-gateways
19712024-10-30T17:51:50.081ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46626
19722024-10-30T17:51:50.082ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 230 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46626 req_id = 07cb64a6-90ca-4142-8ef0-ec5bb8ca7c4e response_code = 200 uri = /v2p
19732024-10-30T17:51:50.095ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52465
19742024-10-30T17:51:50.095ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19752024-10-30T17:51:50.136ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19762024-10-30T17:51:50.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100145 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52465 req_id = c82857ca-e8e7-451f-851d-4f909b743d17 response_code = 204 uri = /switch-ports
19772024-10-30T17:51:50.196ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58305
19782024-10-30T17:51:50.196ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19792024-10-30T17:51:50.196ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 342 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52465 req_id = 6f803791-fc80-4d93-999f-93dcda776a46 response_code = 200 uri = /network-bootstore-config
19802024-10-30T17:51:50.203ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7086 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51369 req_id = 59cc820c-ffe4-478b-b629-7eca47cdfa92 response_code = 204 uri = /vpc-routes
19812024-10-30T17:51:50.203ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7253 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:38604 req_id = fcee71cd-878a-43f4-b9a6-065d5e5d445f response_code = 204 uri = /vpc-routes
19822024-10-30T17:51:50.239ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19832024-10-30T17:51:50.296ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100329 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58305 req_id = ea0f1ea9-5f61-4b42-90af-c3e5e97b506f response_code = 204 uri = /switch-ports
19842024-10-30T17:51:50.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58305 req_id = df5cc0ed-93f0-47bf-b621-07248fad201d response_code = 200 uri = /network-bootstore-config
19852024-10-30T17:52:08.861ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:56410
19862024-10-30T17:52:08.862ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 38 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:56410 req_id = b18fe796-5b1b-4b7e-8425-669a4069d2ad response_code = 200 uri = /baseboard
19872024-10-30T17:52:19.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56934
19882024-10-30T17:52:19.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 120 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56934 req_id = c2d901ce-3ffe-4b77-be33-01e894377e5c response_code = 200 uri = /vpc-routes
19892024-10-30T17:52:19.747ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
19902024-10-30T17:52:19.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56934 req_id = 099548b0-5309-4573-b2a8-833787d40dc9 response_code = 204 uri = /eip-gateways
19912024-10-30T17:52:19.764ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 259 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:56934 req_id = 2b324d3e-de1e-4819-a39f-e322075ad63b response_code = 204 uri = /vpc-routes
19922024-10-30T17:52:19.875ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:47526
19932024-10-30T17:52:19.875ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
19942024-10-30T17:52:19.921ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
19952024-10-30T17:52:19.995ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 120616 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:47526 req_id = 475e9250-1c98-4525-91dc-e1acd279ce8a response_code = 204 uri = /switch-ports
19962024-10-30T17:52:19.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51242
19972024-10-30T17:52:19.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41481
19982024-10-30T17:52:19.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41481 req_id = 3049eb52-4590-40a9-93d1-ff04ace45c24 response_code = 200 uri = /vpc-routes
19992024-10-30T17:52:19.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 214 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51242 req_id = 44ccddba-956d-4d59-8224-58cf9d23ebcd response_code = 200 uri = /v2p
20002024-10-30T17:52:19.996ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 216 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:47526 req_id = 75bdbae5-1744-47f4-8a2e-f4d30abb758f response_code = 200 uri = /network-bootstore-config
20012024-10-30T17:52:20.041ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20022024-10-30T17:52:20.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 152 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41481 req_id = 27a24a98-f5e8-4f98-9312-74fe18c05f4c response_code = 204 uri = /eip-gateways
20032024-10-30T17:52:20.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52516
20042024-10-30T17:52:20.047ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52516 req_id = f7e9a8bc-ac54-4158-85c2-80e86326edd4 response_code = 200 uri = /vpc-routes
20052024-10-30T17:52:20.051ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 202 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41481 req_id = 4dd6c2d3-97b4-46e2-97c8-9544ac3d3529 response_code = 204 uri = /vpc-routes
20062024-10-30T17:52:20.097ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20072024-10-30T17:52:20.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52516 req_id = df920928-bd37-4351-9360-6d61cad947d1 response_code = 204 uri = /eip-gateways
20082024-10-30T17:52:20.098ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42267
20092024-10-30T17:52:20.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42267 req_id = 0cba068f-75c7-4d97-96f9-6e84f3b270da response_code = 200 uri = /v2p
20102024-10-30T17:52:20.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:52516 req_id = 8bdacc15-a048-4437-ab89-353d9c6aa5fc response_code = 204 uri = /vpc-routes
20112024-10-30T17:52:20.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35248
20122024-10-30T17:52:20.139ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20132024-10-30T17:52:20.183ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20142024-10-30T17:52:20.239ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100249 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35248 req_id = 415ac923-5119-4359-aa51-ca991fa4bbb6 response_code = 204 uri = /switch-ports
20152024-10-30T17:52:20.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35248 req_id = 7d2aedab-17c8-4087-87ff-75561277362b response_code = 200 uri = /network-bootstore-config
20162024-10-30T17:52:20.346ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58800
20172024-10-30T17:52:20.347ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58800 req_id = 259f3aae-12f6-4a90-909a-6908af13a63d response_code = 200 uri = /v2p
20182024-10-30T17:52:20.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46274
20192024-10-30T17:52:20.363ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20202024-10-30T17:52:20.397ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20212024-10-30T17:52:20.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93238 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46274 req_id = a1e7a182-4ca0-40fa-a8fb-fe31192160b8 response_code = 204 uri = /switch-ports
20222024-10-30T17:52:20.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46274 req_id = 4a729ca1-d42e-4310-8616-3cfb663d1e82 response_code = 200 uri = /network-bootstore-config
20232024-10-30T17:52:33.105ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
20242024-10-30T17:52:38.894ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45149
20252024-10-30T17:52:38.894ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 33 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45149 req_id = 97f2157f-8b67-46e9-9c74-86c4adefbeb7 response_code = 200 uri = /baseboard
20262024-10-30T17:52:49.633ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48622
20272024-10-30T17:52:49.633ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48622 req_id = 7548e886-0a2c-409a-9837-b9463829de4d response_code = 200 uri = /vpc-routes
20282024-10-30T17:52:49.642ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54459
20292024-10-30T17:52:49.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54459 req_id = b2fa97d5-5dea-4b36-a21b-5e4cd9b83133 response_code = 200 uri = /v2p
20302024-10-30T17:52:49.652ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20312024-10-30T17:52:49.652ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48622 req_id = e548bb20-3c8b-444e-9c5f-f76bd257e117 response_code = 204 uri = /eip-gateways
20322024-10-30T17:52:49.663ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 196 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48622 req_id = 580d9087-d0b1-4a3b-81c5-ebf4ca0f58ea response_code = 204 uri = /vpc-routes
20332024-10-30T17:52:49.748ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59599
20342024-10-30T17:52:49.748ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59599 req_id = 4f1960ad-ed12-4e3f-991b-062e3a09e312 response_code = 200 uri = /vpc-routes
20352024-10-30T17:52:49.801ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20362024-10-30T17:52:49.801ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 63 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59599 req_id = e23bdfc2-c1d2-47d9-b939-e7fa1d70d788 response_code = 204 uri = /eip-gateways
20372024-10-30T17:52:49.818ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51422
20382024-10-30T17:52:49.819ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20392024-10-30T17:52:49.859ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20402024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104010 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:51422 req_id = 63706cfa-f3af-4cb5-9385-972c847f058a response_code = 204 uri = /switch-ports
20412024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46880
20422024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63451
20432024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:62799
20442024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 353 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59599 req_id = 05496818-be3f-4702-8380-895a03e25f3e response_code = 204 uri = /vpc-routes
20452024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51422 req_id = c29f6eea-d709-4094-8db6-af954a89e979 response_code = 200 uri = /network-bootstore-config
20462024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 191 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63451 req_id = f777e83a-db48-4771-a2f8-f68ca54a7b22 response_code = 200 uri = /vpc-routes
20472024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 277 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:62799 req_id = 6c1437af-7354-4ff0-adab-db91262a056a response_code = 200 uri = /v2p
20482024-10-30T17:52:49.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 309 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46880 req_id = 38d9883d-0748-4deb-9d54-b23e69c63a6a response_code = 200 uri = /v2p
20492024-10-30T17:52:49.935ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60932
20502024-10-30T17:52:49.935ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20512024-10-30T17:52:49.974ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20522024-10-30T17:52:50.028ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92997 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60932 req_id = 17c44767-709d-4d17-86df-5b8313117edf response_code = 204 uri = /switch-ports
20532024-10-30T17:52:50.029ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46364
20542024-10-30T17:52:50.029ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20552024-10-30T17:52:50.029ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 170 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63451 req_id = 6cc14303-5964-4b6b-a574-e41ee43a2861 response_code = 204 uri = /eip-gateways
20562024-10-30T17:52:50.029ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 238 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60932 req_id = d373c568-acbe-4732-a58b-88b3fb39ca27 response_code = 200 uri = /network-bootstore-config
20572024-10-30T17:52:50.029ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20582024-10-30T17:52:50.041ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 258 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63451 req_id = eb7540cf-0212-4f0e-a2f3-8fca0273d589 response_code = 204 uri = /vpc-routes
20592024-10-30T17:52:50.071ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20602024-10-30T17:52:50.124ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 95545 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46364 req_id = 40168a96-57a1-48e0-a65c-991924dbf41f response_code = 204 uri = /switch-ports
20612024-10-30T17:52:50.125ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46364 req_id = 1f7b1228-3229-40ee-9fd1-3c560d093885 response_code = 200 uri = /network-bootstore-config
20622024-10-30T17:53:08.931ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52373
20632024-10-30T17:53:08.931ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 34 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52373 req_id = dd49a919-5dee-4352-9278-3efbbf906bae response_code = 200 uri = /baseboard
20642024-10-30T17:53:19.649ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58887
20652024-10-30T17:53:19.649ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58887 req_id = 76148106-c4a4-456f-9aee-cdc7c7572ce8 response_code = 200 uri = /vpc-routes
20662024-10-30T17:53:19.711ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20672024-10-30T17:53:19.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 148 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58887 req_id = 8d793ba7-e241-4a29-b568-8649ed3035a6 response_code = 204 uri = /eip-gateways
20682024-10-30T17:53:19.723ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 240 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58887 req_id = 3af7af4c-33d4-45c3-9f23-36bb54f0f8b5 response_code = 204 uri = /vpc-routes
20692024-10-30T17:53:19.852ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39069
20702024-10-30T17:53:19.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 50 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39069 req_id = ccdf7e11-3381-4b72-8066-e1195c5dbb7f response_code = 200 uri = /vpc-routes
20712024-10-30T17:53:19.901ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20722024-10-30T17:53:19.901ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 77 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39069 req_id = bcd968cc-f81b-4533-b0e5-22e433d25f31 response_code = 204 uri = /eip-gateways
20732024-10-30T17:53:19.930ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46113
20742024-10-30T17:53:19.930ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46113 req_id = 44236e34-2031-45bc-9ed0-38e611557c25 response_code = 200 uri = /v2p
20752024-10-30T17:53:19.933ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40780
20762024-10-30T17:53:19.933ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20772024-10-30T17:53:19.975ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20782024-10-30T17:53:20.047ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114448 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40780 req_id = 739e180f-ba5a-4dcb-bb21-6ccbb9e9c953 response_code = 204 uri = /switch-ports
20792024-10-30T17:53:20.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61694
20802024-10-30T17:53:20.048ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46877
20812024-10-30T17:53:20.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:61694 req_id = e0230ced-2aee-4b0c-9d33-34ccfd2e63b6 response_code = 200 uri = /vpc-routes
20822024-10-30T17:53:20.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 250 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40780 req_id = ae8373ca-e596-4feb-a987-bcad5cd15a10 response_code = 200 uri = /network-bootstore-config
20832024-10-30T17:53:20.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 325 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46877 req_id = 92751ad6-fe2b-4ecd-9616-eefc7a502f40 response_code = 200 uri = /v2p
20842024-10-30T17:53:20.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 484 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39069 req_id = 250e9e04-d37b-4f9e-9e49-fa7726b9746c response_code = 204 uri = /vpc-routes
20852024-10-30T17:53:20.066ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
20862024-10-30T17:53:20.066ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61694 req_id = 612e8997-3e30-44cf-ab0b-2a52b9a0f7b6 response_code = 204 uri = /eip-gateways
20872024-10-30T17:53:20.092ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52080
20882024-10-30T17:53:20.092ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20892024-10-30T17:53:20.134ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20902024-10-30T17:53:20.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99910 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52080 req_id = 11dfc57d-7471-4ca9-9ffb-9fb11a392978 response_code = 204 uri = /switch-ports
20912024-10-30T17:53:20.192ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45907
20922024-10-30T17:53:20.192ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59095
20932024-10-30T17:53:20.193ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
20942024-10-30T17:53:20.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 447 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61694 req_id = f26ae614-63be-4455-9e75-5f8888b8d350 response_code = 204 uri = /vpc-routes
20952024-10-30T17:53:20.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45907 req_id = ceebd40f-50d2-4c2d-bb6b-f8a0ec3f1e05 response_code = 200 uri = /v2p
20962024-10-30T17:53:20.193ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 434 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52080 req_id = bdb43037-87aa-487b-82c0-64f677b4c449 response_code = 200 uri = /network-bootstore-config
20972024-10-30T17:53:20.232ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
20982024-10-30T17:53:20.289ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96001 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59095 req_id = 532a5dea-0b3f-403f-af99-164912cdd030 response_code = 204 uri = /switch-ports
20992024-10-30T17:53:20.289ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 145 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59095 req_id = 1ecef22d-58c1-47f2-927c-989ecf3401e0 response_code = 200 uri = /network-bootstore-config
21002024-10-30T17:53:38.966ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41782
21012024-10-30T17:53:38.966ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 36 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41782 req_id = 2f3d29dd-191a-4dc0-a1a4-70bf25c99396 response_code = 200 uri = /baseboard
21022024-10-30T17:53:42.166ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:39132
21032024-10-30T17:53:42.166ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:39132 req_id = 810e9e5f-c796-4934-a5cf-a634292bb274 response_code = 200 uri = /v2p
21042024-10-30T17:53:42.167ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:838 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(14486778) }
21052024-10-30T17:53:42.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1230 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:39132 req_id = 9d4de26e-6403-4783-9a92-ea9f4a9424ce response_code = 204 uri = /v2p
21062024-10-30T17:53:42.300ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33359
21072024-10-30T17:53:42.306ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:555 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(14486778))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-fc6c9339ab8", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 1a2e25c2-ff32-4584-bf21-4980e98a41c3, read_only_parent: Some(Volume { block_size: 512, id: 96f28598-112b-4fab-baa9-7fb05487f915, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: c11fa797-255b-4f34-887d-6939ca1db224, key: Some("odA042DAzKY9S6NyGSh7XC/V/kGtujUt+i7Qd1ilwKM="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 1a2e25c2-ff32-4584-bf21-4980e98a41c3, key: Some("Otpx1yd3tDmAMT/ruhu2vPKLvf19xyniWxu3FtiyDyE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::19]:19001"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-fc6c9339ab8" }] }), cloud_init_bytes: Some(..) } instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff metadata = InstanceMetadata { silo_id: b0544d6c-57b7-4afa-8a45-50c6d7ea691b, project_id: acea9de3-f019-4404-9fee-854f016edb30 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-30T17:53:42.300158916Z }
21082024-10-30T17:53:42.306ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:588 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff migration_id = None propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21092024-10-30T17:53:42.306ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1271 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff migration_id = None propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(14486778))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: d12ca7ff-9948-4c80-bc07-39036ffabcdf, kind: Instance { id: 264bf454-3e5c-433b-b0fb-bf5fce4db6ff }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(14486778), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-fc6c9339ab8", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 1a2e25c2-ff32-4584-bf21-4980e98a41c3, read_only_parent: Some(Volume { block_size: 512, id: 96f28598-112b-4fab-baa9-7fb05487f915, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: c11fa797-255b-4f34-887d-6939ca1db224, key: Some("odA042DAzKY9S6NyGSh7XC/V/kGtujUt+i7Qd1ilwKM="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::18]:19001", "[fd00:1122:3344:101::14]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 1a2e25c2-ff32-4584-bf21-4980e98a41c3, key: Some("Otpx1yd3tDmAMT/ruhu2vPKLvf19xyniWxu3FtiyDyE="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::12]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::19]:19001"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-fc6c9339ab8" }] }), cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-30T17:53:42.300158916Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
21102024-10-30T17:53:42.307ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6538 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33359 req_id = 908f17ba-f8e8-429e-9bb7-c14b81352461 response_code = 200 uri = /vmms/ee8899a9-023b-4452-bf65-0fe051ddb789
21112024-10-30T17:53:42.416ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61305
21122024-10-30T17:53:42.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61305 req_id = e35ff583-4165-4f62-ab23-88648fec5eb5 response_code = 200 uri = /vmms/ee8899a9-023b-4452-bf65-0fe051ddb789/state
21132024-10-30T17:53:42.418ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:430 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 14486778 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
21142024-10-30T17:53:42.418ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21152024-10-30T17:53:42.459ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789 file = illumos-utils/src/zone.rs:316 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21162024-10-30T17:53:42.480ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789 file = illumos-utils/src/zone.rs:354 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21172024-10-30T17:53:42.493ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55976
21182024-10-30T17:53:42.493ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55976 req_id = 6146b2cb-0aa7-4b68-af82-2bac60fcaab5 response_code = 200 uri = /vpc-routes
21192024-10-30T17:53:42.513ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {d12ca7ff-9948-4c80-bc07-39036ffabcdf: {10.151.2.175: {d0062ed7-1137-4a67-a25a-99967dd89d2f}, 10.151.2.176: {d0062ed7-1137-4a67-a25a-99967dd89d2f}}} }
21202024-10-30T17:53:43.630ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21212024-10-30T17:53:43.630ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789 zone file = illumos-utils/src/running_zone.rs:488 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21222024-10-30T17:53:43.840ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 37.221771ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21232024-10-30T17:53:43.886ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 134.469024ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21242024-10-30T17:53:44.042ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 144.677844ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21252024-10-30T17:53:44.204ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 329.118459ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21262024-10-30T17:53:44.551ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 573.719395ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21272024-10-30T17:53:45.136ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 1.112381234s file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21282024-10-30T17:53:46.261ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 528.057108ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21292024-10-30T17:53:46.801ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 588.285404ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21302024-10-30T17:53:47.400ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 619.116595ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21312024-10-30T17:53:48.031ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 1.038165942s file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 zone = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
21322024-10-30T17:53:49.100ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789 file = sled-agent/src/instance.rs:1777 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21332024-10-30T17:53:49.110ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 63.86528ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21342024-10-30T17:53:49.183ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 60.6219ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21352024-10-30T17:53:49.253ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789") failed: Property not found. retry in 179.485734ms file = illumos-utils/src/svc.rs:36 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21362024-10-30T17:53:49.448ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1786 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21372024-10-30T17:53:49.488ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1810 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21382024-10-30T17:53:49.605ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41223
21392024-10-30T17:53:49.624ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40625
21402024-10-30T17:53:49.624ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40625 req_id = 6590ba67-0ffe-4f95-8a21-0ed7a7ca9741 response_code = 200 uri = /vpc-routes
21412024-10-30T17:53:49.642ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {d12ca7ff-9948-4c80-bc07-39036ffabcdf: {10.151.2.176: {d0062ed7-1137-4a67-a25a-99967dd89d2f}, 10.151.2.175: {d0062ed7-1137-4a67-a25a-99967dd89d2f}}} }
21422024-10-30T17:53:49.642ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40625 req_id = be0b6cac-ffe0-40e7-86a7-9fb1743b1e6e response_code = 204 uri = /eip-gateways
21432024-10-30T17:53:49.658ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:926 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21442024-10-30T17:53:49.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52265 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41223 req_id = bb18e6dd-b1da-4eb1-8021-2219913828bc response_code = 200 uri = /vmms/ee8899a9-023b-4452-bf65-0fe051ddb789/state
21452024-10-30T17:53:49.659ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1215 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
21462024-10-30T17:53:49.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7147126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55976 req_id = 955fdaf5-73b3-4423-9ef2-0ff735364f14 response_code = 204 uri = /eip-gateways
21472024-10-30T17:53:49.660ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-10-30T17:53:49.660703939Z } file = sled-agent/src/instance.rs:776 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21482024-10-30T17:53:49.660ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:798 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-30T17:53:49.660703939Z } propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21492024-10-30T17:53:49.660ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:696 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-30T17:53:49.660703939Z }, migration_in: None, migration_out: None }
21502024-10-30T17:53:49.836ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39460
21512024-10-30T17:53:49.836ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39460 req_id = aad7052d-1978-4ae1-97e0-63b76beb5f81 response_code = 200 uri = /v2p
21522024-10-30T17:53:49.847ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54687
21532024-10-30T17:53:49.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 336 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:40625 req_id = 87d4ce34-8fd8-4764-b47a-051a421b6392 response_code = 204 uri = /vpc-routes
21542024-10-30T17:53:49.895ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38111
21552024-10-30T17:53:49.898ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64302
21562024-10-30T17:53:49.899ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 40 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64302 req_id = be7d9320-5634-483a-b49b-4012e5c5e3d1 response_code = 200 uri = /vpc-routes
21572024-10-30T17:53:49.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7997 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38111 req_id = 6e5c4f3e-06ad-4658-8911-be912e326240 response_code = 200 uri = /vmms/ee8899a9-023b-4452-bf65-0fe051ddb789/state
21582024-10-30T17:53:49.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 55404 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54687 req_id = 6555fd1b-d2bd-4aef-863a-787b82817614 response_code = 200 uri = /vmms/ee8899a9-023b-4452-bf65-0fe051ddb789/state
21592024-10-30T17:53:49.903ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-10-30T17:53:49.903637277Z } file = sled-agent/src/instance.rs:776 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21602024-10-30T17:53:49.903ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:798 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-30T17:53:49.903637277Z } propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21612024-10-30T17:53:49.903ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:696 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-30T17:53:49.903637277Z }, migration_in: None, migration_out: None }
21622024-10-30T17:53:49.920ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {d12ca7ff-9948-4c80-bc07-39036ffabcdf: {10.151.2.175: {d0062ed7-1137-4a67-a25a-99967dd89d2f}, 10.151.2.176: {d0062ed7-1137-4a67-a25a-99967dd89d2f}}} }
21632024-10-30T17:53:49.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64302 req_id = 0f06450e-dca5-421f-8529-01eab0c2521e response_code = 204 uri = /eip-gateways
21642024-10-30T17:53:49.972ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46443
21652024-10-30T17:53:49.972ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
21662024-10-30T17:53:50.016ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
21672024-10-30T17:53:50.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110199 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:46443 req_id = 6a9cb378-ba40-4aab-8ed9-8e901ccad821 response_code = 204 uri = /switch-ports
21682024-10-30T17:53:50.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54094
21692024-10-30T17:53:50.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45360
21702024-10-30T17:53:50.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53635
21712024-10-30T17:53:50.083ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42168
21722024-10-30T17:53:50.083ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
21732024-10-30T17:53:50.087ZINFOSledAgent (PortManager): skipping delta compute for subnet file = illumos-utils/src/opte/port_manager.rs:469 new_vers = RouterVersion { router_id: a8b55317-fe26-46b3-9808-fb373777f7fa, version: 3 } old_vers = RouterVersion { router_id: a8b55317-fe26-46b3-9808-fb373777f7fa, version: 3 } subnet = RouterId { vni: Vni(14486778), kind: System }
21742024-10-30T17:53:50.087ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4433 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55976 req_id = 9861b799-e9b2-44bd-84fb-d664c3f492f8 response_code = 204 uri = /vpc-routes
21752024-10-30T17:53:50.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4394 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54094 req_id = c6ec1395-6a2f-45ac-a7e2-1d7a19193b8f response_code = 200 uri = /v2p
21762024-10-30T17:53:50.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4760 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64302 req_id = cdab359e-392b-45ab-9e5c-2909cc2ce60f response_code = 204 uri = /vpc-routes
21772024-10-30T17:53:50.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4609 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46443 req_id = 9ca1f252-2721-4874-aa43-91c41b8e8481 response_code = 200 uri = /network-bootstore-config
21782024-10-30T17:53:50.088ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4548 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42168 req_id = cb929f1f-25de-4adb-b94e-0ad627270946 response_code = 200 uri = /v2p
21792024-10-30T17:53:50.123ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
21802024-10-30T17:53:50.130ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55781
21812024-10-30T17:53:50.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 79 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55781 req_id = d040ae82-2c8f-4dbb-bfaf-aaef5a1d4c5b response_code = 200 uri = /vpc-routes
21822024-10-30T17:53:50.173ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {d12ca7ff-9948-4c80-bc07-39036ffabcdf: {10.151.2.176: {d0062ed7-1137-4a67-a25a-99967dd89d2f}, 10.151.2.175: {d0062ed7-1137-4a67-a25a-99967dd89d2f}}} }
21832024-10-30T17:53:50.174ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 736 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55781 req_id = 6eb0ad09-2751-4589-86e3-133ddfe3e2d7 response_code = 204 uri = /eip-gateways
21842024-10-30T17:53:50.182ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4080
21852024-10-30T17:53:50.182ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98862 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:45360 req_id = 10612ad5-ed63-4365-99d8-da4f73535a84 response_code = 204 uri = /switch-ports
21862024-10-30T17:53:50.186ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 735 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45360 req_id = d3805ba7-e06f-479a-9f82-22f48291b2de response_code = 200 uri = /network-bootstore-config
21872024-10-30T17:53:50.192ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 222 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55781 req_id = e650a0a1-6b8a-4fa1-b8d0-430212a7da32 response_code = 204 uri = /vpc-routes
21882024-10-30T17:53:50.222ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4104
21892024-10-30T17:53:50.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 198478 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53635 req_id = d3c3bc74-d743-4dbb-971f-b08d731c2391 response_code = 204 uri = /switch-ports
21902024-10-30T17:53:50.282ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53635 req_id = 5e6fc8c0-72c8-4e81-b6c2-78eea5e4a5f4 response_code = 200 uri = /network-bootstore-config
21912024-10-30T17:53:59.946ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61827
21922024-10-30T17:53:59.947ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 552 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61827 req_id = 5b9ef422-1c5e-4f71-98f1-7a5edb26e540 response_code = 200 uri = /vmms/ee8899a9-023b-4452-bf65-0fe051ddb789/state
21932024-10-30T17:53:59.947ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-10-30T17:53:59.947205458Z } file = sled-agent/src/instance.rs:776 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21942024-10-30T17:53:59.947ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:798 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-30T17:53:59.947205458Z } propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21952024-10-30T17:53:59.947ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:696 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-30T17:53:59.947205458Z }, migration_in: None, migration_out: None }
21962024-10-30T17:53:59.985ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-10-30T17:53:59.985079581Z } file = sled-agent/src/instance.rs:776 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21972024-10-30T17:53:59.985ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:798 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-10-30T17:53:59.985079581Z } propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21982024-10-30T17:53:59.985ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:812 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
21992024-10-30T17:53:59.985ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22002024-10-30T17:53:59.986ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 3483541642628734897 n_samples = 6 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
22012024-10-30T17:53:59.986ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 2329747649594060562 n_samples = 6 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
22022024-10-30T17:53:59.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
22032024-10-30T17:53:59.987ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/3485ccb7-7923-4269-8b7b-16caa41fa8d8/debug/bundle/zone", "/pool/int/adb893fc-56e4-4ac7-a502-4c948ee9f9bd/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/5c811413-1d4e-403e-b8bb-331b2268c476/crypt/debug", "/pool/ext/7c337fba-b737-4f90-a327-065b4f3b4476/crypt/debug", "/pool/ext/46e37abb-39ab-4652-aa9d-61a2079024f1/crypt/debug", "/pool/ext/8b4477e5-c445-4846-a713-78ddac5696a7/crypt/debug", "/pool/ext/09d27db0-c064-443e-8e36-e29bb4d8f58d/crypt/debug", "/pool/ext/9fecadf7-1f9f-4c05-baeb-3ca812eb625c/crypt/debug", "/pool/ext/c84ed810-b8fd-4283-8511-603d43b629a7/crypt/debug", "/pool/ext/f6c51041-d1f1-4026-854a-17c6531b176e/crypt/debug", "/pool/ext/b47c81a2-215a-4a96-ada9-ce831b19ee43/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789
22042024-10-30T17:54:00.627ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789", bundle_id: 5a90bb66-da35-4fb2-9718-76736e8b5d85 }, time_created: 2024-10-30T17:53:59.988809561Z, version: 0, cause: TerminatedInstance }
22052024-10-30T17:54:00.627ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_ee8899a9-023b-4452-bf65-0fe051ddb789 file = sled-agent/src/instance.rs:1026 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
22062024-10-30T17:54:02.396ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
22072024-10-30T17:54:02.412ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(14486778), kind: System }
22082024-10-30T17:54:02.412ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:964 id = RouterId { vni: Vni(14486778), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
22092024-10-30T17:54:02.413ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:696 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-30T17:54:02.413113188Z }, migration_in: None, migration_out: None }
22102024-10-30T17:54:02.438ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:696 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-30T17:54:02.413113188Z }, migration_in: None, migration_out: None }
22112024-10-30T17:54:02.441ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:963 instance_id = 264bf454-3e5c-433b-b0fb-bf5fce4db6ff propolis_id = ee8899a9-023b-4452-bf65-0fe051ddb789
22122024-10-30T17:54:02.834ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55417
22132024-10-30T17:54:02.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55417 req_id = 4a988b30-bf04-466d-bc50-a228be1ef499 response_code = 200 uri = /vpc-routes
22142024-10-30T17:54:02.885ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {d12ca7ff-9948-4c80-bc07-39036ffabcdf: {10.151.2.175: {d0062ed7-1137-4a67-a25a-99967dd89d2f}, 10.151.2.176: {d0062ed7-1137-4a67-a25a-99967dd89d2f}}} }
22152024-10-30T17:54:02.885ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55417 req_id = 37bee9e9-6849-440f-9ff5-017e77173328 response_code = 204 uri = /eip-gateways
22162024-10-30T17:54:02.898ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55417 req_id = 7c2b126f-4c1b-423a-97e7-07e6f1e4556b response_code = 204 uri = /vpc-routes
22172024-10-30T17:54:03.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43014
22182024-10-30T17:54:03.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43014 req_id = 193494a8-a565-4dff-b403-376c57de46e4 response_code = 200 uri = /v2p
22192024-10-30T17:54:03.295ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46970
22202024-10-30T17:54:03.295ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46970 req_id = 6cf18043-0fa1-4376-b4a2-227eef471a2c response_code = 200 uri = /v2p
22212024-10-30T17:54:03.455ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54261
22222024-10-30T17:54:03.456ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 62 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54261 req_id = ce5ca3ff-8433-4cb0-b50f-e91aee0aa035 response_code = 200 uri = /vpc-routes
22232024-10-30T17:54:03.468ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
22242024-10-30T17:54:03.468ZINFOSledAgent: IGW mapping changed; external IPs refreshed file = sled-agent/src/sled_agent.rs:1215 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273
22252024-10-30T17:54:03.468ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54261 req_id = c2ffae4a-5465-4222-8d81-d304831b8d74 response_code = 204 uri = /eip-gateways
22262024-10-30T17:54:03.475ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 228 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54261 req_id = eaf4706b-aa8b-4fc2-85ab-3800b5c00dc7 response_code = 204 uri = /vpc-routes
22272024-10-30T17:54:04.047ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41732
22282024-10-30T17:54:04.048ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41732 req_id = f8cb7426-53b4-48bf-a4fe-85f7f1c7919b response_code = 200 uri = /vpc-routes
22292024-10-30T17:54:04.061ZINFOSledAgent: IGW mapping received file = sled-agent/src/sled_agent.rs:1189 sled_id = 297da6c0-3dc5-4f20-9a57-48a339550273 values = ExternalIpGatewayMap { mappings: {} }
22302024-10-30T17:54:04.061ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41732 req_id = 724dc4d3-4c26-4055-a22d-ee02b8b5399e response_code = 204 uri = /eip-gateways
22312024-10-30T17:54:04.069ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 188 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41732 req_id = 514343f9-38a4-420c-b09c-8aacbed13d88 response_code = 204 uri = /vpc-routes