Max level shown:
1 [ Oct 4 22:11:49 Enabled. ]
2 [ Oct 4 22:11:49 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Oct 4 22:11:49 Method "start" exited with status 0. ]
4 [ Oct 4 22:11:49 Rereading configuration. ]
5 [ Oct 4 22:11:49 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-10-04T22:11:49.167ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-10-04T22:11:49.167ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-10-04T22:11:49.221ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-10-04T22:11:49.221ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-10-04T22:11:49.335ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-10-04T22:11:49.344ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-10-04T22:11:49.353ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-10-04T22:11:49.959ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-10-04T22:11:50.095ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-10-04T22:11:50.096ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-10-04T22:11:50.096ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-10-04T22:11:50.096ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-10-04T22:11:50.096ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-10-04T22:11:50.096ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:717
212024-10-04T22:11:50.099ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:624
222024-10-04T22:11:50.099ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:766
232024-10-04T22:11:50.099ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-10-04T22:11:50.099ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:825
252024-10-04T22:11:50.099ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-10-04T22:11:50.099ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-10-04T22:11:50.099ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-10-04T22:11:50.099ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-10-04T22:11:50.099ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-10-04T22:11:50.110ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:367
312024-10-04T22:11:50.110ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool)
322024-10-04T22:11:50.207ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal }
332024-10-04T22:11:50.615ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-10-04T22:11:50.615ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-10-04T22:11:50.615ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-10-04T22:11:50.615ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-10-04T22:11:50.615ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-10-04T22:11:50.615ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-10-04T22:11:50.615ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-10-04T22:11:50.624ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:367
412024-10-04T22:11:50.624ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:378 id = 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool)
422024-10-04T22:11:50.712ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal }
432024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
522024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
532024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
542024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
752024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
762024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
772024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
782024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
812024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
822024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
832024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
842024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
852024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
892024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
902024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-10-04T22:11:51.130ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-10-04T22:11:51.130ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-10-04T22:11:51.130ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-10-04T22:11:51.130ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-10-04T22:11:51.130ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1002024-10-04T22:11:51.130ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1012024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1022024-10-04T22:11:51.130ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1032024-10-04T22:11:51.130ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-10-04T22:11:51.130ZINFOSledAgent (bootstore): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-10-04T22:11:51.130ZINFOSledAgent (bootstore): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-10-04T22:11:51.130ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-10-04T22:11:51.130ZINFOSledAgent (bootstore): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-10-04T22:11:51.130ZINFOSledAgent (bootstore): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-10-04T22:11:51.130ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-10-04T22:11:51.130ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-10-04T22:11:51.130ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-10-04T22:11:51.141ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:739
1132024-10-04T22:11:51.141ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 1309, tv_nsec: 1804055 } time_to_next_cleanup = 599.999942599s
1142024-10-04T22:11:51.142ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3924
1152024-10-04T22:11:51.142ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-10-04T22:11:51.145ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-10-04T22:11:51.145ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-10-04T22:11:51.145ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-10-04T22:11:51.147ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:51 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-10-04T22:11:51.147ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1074
1212024-10-04T22:11:51.147ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72
1222024-10-04T22:11:51.147ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-10-04T22:11:51.147ZINFOSledAgent (RSS): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-10-04T22:11:51.147ZINFOSledAgent (RSS): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-10-04T22:11:51.147ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1137
1262024-10-04T22:11:51.147ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-10-04T22:11:51.147ZINFOSledAgent (RSS): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-10-04T22:11:51.147ZINFOSledAgent (RSS): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-10-04T22:11:51.147ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1179
1302024-10-04T22:11:51.147ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-10-04T22:11:51.147ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-10-04T22:11:51.147ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-10-04T22:11:51.147ZINFOSledAgent (RSS): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-10-04T22:11:51.147ZINFOSledAgent (RSS): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-10-04T22:11:51.148ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 4cb5ec19-6817-424a-9a2f-064289331fa0, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 544b0af3-4a18-4467-8697-af690997796e (sled), rack_id: 4cb5ec19-6817-424a-9a2f-064289331fa0, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-10-04T22:11:51.148ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1213
1372024-10-04T22:11:51.148ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-10-04T22:11:51.148ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-10-04T22:11:51.148ZINFOSledAgent (bootstore): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-10-04T22:11:51.148ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-10-04T22:11:51.148ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:232 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 544b0af3-4a18-4467-8697-af690997796e (sled), rack_id: 4cb5ec19-6817-424a-9a2f-064289331fa0, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-10-04T22:11:51.149ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4221
1432024-10-04T22:11:51.149ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4603
1442024-10-04T22:11:51.153ZINFOSledAgent (SledAgentSprocketsClient): Loaded keys and certs BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/client.rs:50
1452024-10-04T22:11:51.159ZINFOSledAgent (SledAgentSprocketsClient): Certificate chain verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1462024-10-04T22:11:51.159ZINFOSledAgent (SledAgentSprocketsClient): Signature verified successfully BootstrapAgentClient = [fdb0:18c0:4d0d:a02a::1]:12346 file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1472024-10-04T22:11:51.163ZINFOSledAgent (SledAgentSprocketsServer): Certificate chain verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:307
1482024-10-04T22:11:51.163ZINFOSledAgent (SledAgentSprocketsServer): Signature verified successfully file = /home/build/.cargo/git/checkouts/sprockets-82488b5b61bf00db/cc13773/tls/src/keys.rs:376
1492024-10-04T22:11:51.163ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:74 remote_addr = [fdb0:18c0:4d0d:a02a::1]:36567
1502024-10-04T22:11:51.164ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: 544b0af3-4a18-4467-8697-af690997796e (sled), rack_id: 4cb5ec19-6817-424a-9a2f-064289331fa0, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:362
1512024-10-04T22:11:51.164ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:377
1522024-10-04T22:11:51.164ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1532024-10-04T22:11:51.164ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72
1542024-10-04T22:11:51.164ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1552024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1562024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1572024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1582024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1592024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1602024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1612024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1622024-10-04T22:11:51.164ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1632024-10-04T22:11:51.164ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1642024-10-04T22:11:51.181ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1652024-10-04T22:11:51.203ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1662024-10-04T22:11:51.220ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1672024-10-04T22:11:51.225ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
1682024-10-04T22:11:51.225ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1692024-10-04T22:11:51.225ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:403 sled_id = 544b0af3-4a18-4467-8697-af690997796e
1702024-10-04T22:11:51.225ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1712024-10-04T22:11:51.228ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 61.463837ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e96a04ed-5668-47a0-a235-82255f8af08f", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e96a04ed-5668-47a0-a235-82255f8af08f" } file = clients/ddm-admin-client/src/lib.rs:177
1722024-10-04T22:11:51.238ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1732024-10-04T22:11:51.261ZINFOSledAgent: successfully created encrypted zvol: name="oxi_f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1742024-10-04T22:11:51.261ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/swap" file = sled-agent/src/swap_device.rs:96
1752024-10-04T22:11:51.266ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:418 sled_id = 544b0af3-4a18-4467-8697-af690997796e
1762024-10-04T22:11:51.266ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:128
1772024-10-04T22:11:51.266ZINFOSledAgent (BackingFs): Ensuring dataset oxi_f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/backing/fmd file = sled-agent/src/backing_fs.rs:138
1782024-10-04T22:11:51.291ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1792024-10-04T22:11:51.291ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 169.928165ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ff3c6125-0dec-4a27-9db6-cd2e66412022", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ff3c6125-0dec-4a27-9db6-cd2e66412022" } file = clients/ddm-admin-client/src/lib.rs:177
1802024-10-04T22:11:51.356ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:173
1812024-10-04T22:11:51.463ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1822024-10-04T22:11:51.464ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 168.115147ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a054f08b-6680-4df0-864d-f8f1d793e23d", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a054f08b-6680-4df0-864d-f8f1d793e23d" } file = clients/ddm-admin-client/src/lib.rs:177
1832024-10-04T22:11:51.575ZINFOSledAgent (BackingFs): Mounting oxi_f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:181
1842024-10-04T22:11:51.589ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:189
1852024-10-04T22:11:51.589ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:189
1862024-10-04T22:11:51.589ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:189
1872024-10-04T22:11:51.589ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:195
1882024-10-04T22:11:51.633ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1892024-10-04T22:11:51.634ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 146.328658ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6662d817-5d6d-4887-b630-574864bc98f5", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6662d817-5d6d-4887-b630-574864bc98f5" } file = clients/ddm-admin-client/src/lib.rs:177
1902024-10-04T22:11:51.781ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1912024-10-04T22:11:51.781ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 262.159563ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6bddb201-86e9-4894-832e-858c7b26ab2e", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6bddb201-86e9-4894-832e-858c7b26ab2e" } file = clients/ddm-admin-client/src/lib.rs:177
1922024-10-04T22:11:52.045ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1932024-10-04T22:11:52.045ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 154.874218ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5e24419b-6dee-4e0b-b8ff-5428dd9e0caf", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5e24419b-6dee-4e0b-b8ff-5428dd9e0caf" } file = clients/ddm-admin-client/src/lib.rs:177
1942024-10-04T22:11:52.201ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1952024-10-04T22:11:52.202ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 240.790032ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "690adad3-05d5-49c6-b63d-060ffed71d44", "content-length": "124", "date": "Fri, 04 Oct 2024 22:11:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "690adad3-05d5-49c6-b63d-060ffed71d44" } file = clients/ddm-admin-client/src/lib.rs:177
1962024-10-04T22:11:52.317ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:93 sled_id = 544b0af3-4a18-4467-8697-af690997796e
1972024-10-04T22:11:52.323ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:50448 sled_id = 544b0af3-4a18-4467-8697-af690997796e
1982024-10-04T22:11:52.323ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72 sled_id = 544b0af3-4a18-4467-8697-af690997796e
1992024-10-04T22:11:52.323ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:50448 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = 544b0af3-4a18-4467-8697-af690997796e sled_id = 544b0af3-4a18-4467-8697-af690997796e
2002024-10-04T22:11:52.324ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 175.493006ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2012024-10-04T22:11:52.325ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2022024-10-04T22:11:52.325ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2032024-10-04T22:11:52.325ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:923 underlay_address = fd00:1122:3344:101::1
2042024-10-04T22:11:52.325ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = internal-dns/src/resolver.rs:72
2052024-10-04T22:11:52.444ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
2062024-10-04T22:11:52.501ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 525.8716ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2072024-10-04T22:11:53.029ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.444622725s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2082024-10-04T22:11:54.476ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.73395638s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2092024-10-04T22:11:54.882ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2679
2102024-10-04T22:11:54.882ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2897
2112024-10-04T22:11:54.882ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2549
2122024-10-04T22:11:54.882ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2596
2132024-10-04T22:11:54.882ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2978
2142024-10-04T22:11:54.882ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:3020
2152024-10-04T22:11:54.884ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2822
2162024-10-04T22:11:54.885ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2590
2172024-10-04T22:11:54.886ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='424e4c9a-473d-43a4-aed1-77de951357f7'/> <propval type="astring" name="rack_id" value='4cb5ec19-6817-424a-9a2f-064289331fa0'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fec8:b332'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='544b0af3-4a18-4467-8697-af690997796e'/> <propval type="astring" name="rack_id" value='4cb5ec19-6817-424a-9a2f-064289331fa0'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='544b0af3-4a18-4467-8697-af690997796e'/> <propval type="astring" name="rack_uuid" value='4cb5ec19-6817-424a-9a2f-064289331fa0'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='544b0af3-4a18-4467-8697-af690997796e'/> <propval type="astring" name="rack_uuid" value='4cb5ec19-6817-424a-9a2f-064289331fa0'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2182024-10-04T22:11:54.886ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2192024-10-04T22:11:55.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 61.180588ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-10-04T22:11:55.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 128.899649ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-10-04T22:11:55.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 258.72561ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2222024-10-04T22:11:55.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 497.471178ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2232024-10-04T22:11:56.212ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 4.189174384s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2242024-10-04T22:11:56.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.175947194s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2252024-10-04T22:11:57.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.316877965s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2262024-10-04T22:11:59.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.34909664s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2272024-10-04T22:12:00.403ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 5.664876388s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2282024-10-04T22:12:00.914ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:197 local_addr = [fd00:1122:3344:101::1]:12345
2292024-10-04T22:12:00.914ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:432
2302024-10-04T22:12:00.914ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2312024-10-04T22:12:00.914ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2322024-10-04T22:12:00.915ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2332024-10-04T22:12:00.915ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2342024-10-04T22:12:00.915ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2352024-10-04T22:12:00.915ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3924
2362024-10-04T22:12:00.915ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:81 remote_addr = [fdb0:18c0:4d0d:a02a::1]:36567
2372024-10-04T22:12:01.174ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:117 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2382024-10-04T22:12:01.174ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:252 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2392024-10-04T22:12:01.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2402024-10-04T22:12:01.174ZINFOSledAgent (RSS): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2412024-10-04T22:12:01.174ZINFOSledAgent (RSS): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2422024-10-04T22:12:01.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2432024-10-04T22:12:01.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-10-04T22:12:01.174ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2452024-10-04T22:12:01.179ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4245 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2462024-10-04T22:12:01.179ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4261
2472024-10-04T22:12:01.179ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2482024-10-04T22:12:01.438ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:56301
2492024-10-04T22:12:01.438ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2502024-10-04T22:12:01.439ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 310 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:56301 req_id = 09612c88-2a2a-4b51-91ff-c45d7692b117 response_code = 200 uri = /inventory
2512024-10-04T22:12:01.468ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34374
2522024-10-04T22:12:01.468ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34374 req_id = a72c0897-455c-4b45-a25c-b9efadba63ae response_code = 200 uri = /sled-role
2532024-10-04T22:12:01.470ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2542024-10-04T22:12:01.470ZINFOSledAgent (RSS): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2552024-10-04T22:12:01.470ZINFOSledAgent (RSS): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2562024-10-04T22:12:01.471ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:989
2572024-10-04T22:12:01.503ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: ad44021b-80a8-4e3f-8169-a4f2e44077f0, pool_id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 9daf550a-d979-45cb-b194-42373e18a7b6, pool_id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: b14533ea-b12b-4a13-bb78-43de07b4c715, pool_id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 7393c897-65b9-40bb-a6f1-d76ddbfb0323, pool_id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: e58d30fa-1f17-43a1-94bf-fb29cd4c6d4a, pool_id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 7d0427d9-337b-4e76-914a-2a680eeaa6dd, pool_id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 7c0729de-6bd9-48e1-a6ba-e0c6770fc064, pool_id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 63ea76c8-a8fc-4842-98cf-0bc76c347d60, pool_id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 5e9f88f0-233c-4823-be26-7249b56340a8, pool_id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool) }] } file = sled-agent/src/rack_setup/service.rs:375 sled_address = [fd00:1122:3344:101::1]:12345
2582024-10-04T22:12:01.503ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55039
2592024-10-04T22:12:01.503ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:864 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2602024-10-04T22:12:01.503ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: ad44021b-80a8-4e3f-8169-a4f2e44077f0, pool_id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 9daf550a-d979-45cb-b194-42373e18a7b6, pool_id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: b14533ea-b12b-4a13-bb78-43de07b4c715, pool_id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 7393c897-65b9-40bb-a6f1-d76ddbfb0323, pool_id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: e58d30fa-1f17-43a1-94bf-fb29cd4c6d4a, pool_id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 7d0427d9-337b-4e76-914a-2a680eeaa6dd, pool_id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 7c0729de-6bd9-48e1-a6ba-e0c6770fc064, pool_id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 63ea76c8-a8fc-4842-98cf-0bc76c347d60, pool_id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 5e9f88f0-233c-4823-be26-7249b56340a8, pool_id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2612024-10-04T22:12:01.503ZINFOSledAgent (StorageManager): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2622024-10-04T22:12:01.503ZINFOSledAgent (StorageManager): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2632024-10-04T22:12:01.503ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2642024-10-04T22:12:01.503ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2652024-10-04T22:12:01.503ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2662024-10-04T22:12:01.503ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2672024-10-04T22:12:01.503ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2682024-10-04T22:12:01.659ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:367
2692024-10-04T22:12:01.659ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 676a3716-3d3e-465b-9699-cf98062dabca (zpool)
2702024-10-04T22:12:01.918ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2712024-10-04T22:12:02.406ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2329353703858118291 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2722024-10-04T22:12:02.407ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9526312079148586375 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2732024-10-04T22:12:02.867ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2742024-10-04T22:12:02.898ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }
2752024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:217
2762024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:219
2772024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:223
2782024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:225
2792024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2802024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2812024-10-04T22:12:02.906ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
2822024-10-04T22:12:02.919ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
2832024-10-04T22:12:02.979ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2842024-10-04T22:12:02.987ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_676a3716-3d3e-465b-9699-cf98062dabca/crypt/zone file = sled-storage/src/dataset.rs:280
2852024-10-04T22:12:03.298ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_676a3716-3d3e-465b-9699-cf98062dabca file = sled-storage/src/dataset.rs:366
2862024-10-04T22:12:03.307ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2872024-10-04T22:12:03.307ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2882024-10-04T22:12:03.307ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2892024-10-04T22:12:03.307ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2902024-10-04T22:12:03.307ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2912024-10-04T22:12:03.317ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:367
2922024-10-04T22:12:03.317ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 9af95e51-a064-462a-bea1-8d3042c749de (zpool)
2932024-10-04T22:12:03.498ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }
2942024-10-04T22:12:03.505ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:217
2952024-10-04T22:12:03.506ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:219
2962024-10-04T22:12:03.506ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:223
2972024-10-04T22:12:03.506ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:225
2982024-10-04T22:12:03.506ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2992024-10-04T22:12:03.506ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
3002024-10-04T22:12:03.506ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3012024-10-04T22:12:03.574ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3022024-10-04T22:12:03.582ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9af95e51-a064-462a-bea1-8d3042c749de/crypt/zone file = sled-storage/src/dataset.rs:280
3032024-10-04T22:12:03.901ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9af95e51-a064-462a-bea1-8d3042c749de file = sled-storage/src/dataset.rs:366
3042024-10-04T22:12:03.908ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
3052024-10-04T22:12:03.908ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
3062024-10-04T22:12:03.908ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
3072024-10-04T22:12:03.908ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3082024-10-04T22:12:03.909ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3092024-10-04T22:12:03.918ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:367
3102024-10-04T22:12:03.918ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool)
3112024-10-04T22:12:04.154ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }
3122024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:217
3132024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:219
3142024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:223
3152024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:225
3162024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3172024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3182024-10-04T22:12:04.161ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3192024-10-04T22:12:04.165ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
3202024-10-04T22:12:04.222ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3212024-10-04T22:12:04.229ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_9e3d11a1-af30-407b-b05f-d5f1a5762d41/crypt/zone file = sled-storage/src/dataset.rs:280
3222024-10-04T22:12:04.490ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_9e3d11a1-af30-407b-b05f-d5f1a5762d41 file = sled-storage/src/dataset.rs:366
3232024-10-04T22:12:04.497ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3242024-10-04T22:12:04.497ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3252024-10-04T22:12:04.497ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3262024-10-04T22:12:04.497ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3272024-10-04T22:12:04.497ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3282024-10-04T22:12:04.507ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:367
3292024-10-04T22:12:04.507ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool)
3302024-10-04T22:12:04.643ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }
3312024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:217
3322024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:219
3332024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:223
3342024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:225
3352024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3362024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3372024-10-04T22:12:04.653ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3382024-10-04T22:12:04.716ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3392024-10-04T22:12:04.724ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_ac4d5d54-7085-42e4-a0d6-359bebb3c91f/crypt/zone file = sled-storage/src/dataset.rs:280
3402024-10-04T22:12:04.996ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_ac4d5d54-7085-42e4-a0d6-359bebb3c91f file = sled-storage/src/dataset.rs:366
3412024-10-04T22:12:05.005ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3422024-10-04T22:12:05.005ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3432024-10-04T22:12:05.005ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3442024-10-04T22:12:05.005ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3452024-10-04T22:12:05.005ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3462024-10-04T22:12:05.014ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:367
3472024-10-04T22:12:05.014ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = c4a31058-0731-4581-8d9d-c5c4b66770da (zpool)
3482024-10-04T22:12:05.202ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }
3492024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:217
3502024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:219
3512024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:223
3522024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:225
3532024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3542024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3552024-10-04T22:12:05.209ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3562024-10-04T22:12:05.216ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
3572024-10-04T22:12:05.273ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3582024-10-04T22:12:05.280ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_c4a31058-0731-4581-8d9d-c5c4b66770da/crypt/zone file = sled-storage/src/dataset.rs:280
3592024-10-04T22:12:05.531ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_c4a31058-0731-4581-8d9d-c5c4b66770da file = sled-storage/src/dataset.rs:366
3602024-10-04T22:12:05.538ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3612024-10-04T22:12:05.538ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3622024-10-04T22:12:05.538ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3632024-10-04T22:12:05.538ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3642024-10-04T22:12:05.538ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3652024-10-04T22:12:05.548ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:367
3662024-10-04T22:12:05.548ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 75662bd1-8970-44ab-88a5-02e715476179 (zpool)
3672024-10-04T22:12:05.716ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External }
3682024-10-04T22:12:05.724ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:217
3692024-10-04T22:12:05.724ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:219
3702024-10-04T22:12:05.724ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:223
3712024-10-04T22:12:05.724ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:225
3722024-10-04T22:12:05.724ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3732024-10-04T22:12:05.725ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3742024-10-04T22:12:05.725ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3752024-10-04T22:12:05.779ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3762024-10-04T22:12:05.787ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_75662bd1-8970-44ab-88a5-02e715476179/crypt/zone file = sled-storage/src/dataset.rs:280
3772024-10-04T22:12:06.061ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_75662bd1-8970-44ab-88a5-02e715476179 file = sled-storage/src/dataset.rs:366
3782024-10-04T22:12:06.070ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3792024-10-04T22:12:06.070ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3802024-10-04T22:12:06.070ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3812024-10-04T22:12:06.070ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3822024-10-04T22:12:06.070ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3832024-10-04T22:12:06.073ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 11.320949496s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = 544b0af3-4a18-4467-8697-af690997796e
3842024-10-04T22:12:06.079ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:367
3852024-10-04T22:12:06.079ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool)
3862024-10-04T22:12:06.214ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool), kind: External }
3872024-10-04T22:12:06.218ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
3882024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:217
3892024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:219
3902024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:223
3912024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:225
3922024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3932024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3942024-10-04T22:12:06.222ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
3952024-10-04T22:12:06.281ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3962024-10-04T22:12:06.288ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1279c2fc-c95f-45e2-a67c-783305b4023a/crypt/zone file = sled-storage/src/dataset.rs:280
3972024-10-04T22:12:06.557ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1279c2fc-c95f-45e2-a67c-783305b4023a file = sled-storage/src/dataset.rs:366
3982024-10-04T22:12:06.565ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3992024-10-04T22:12:06.565ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
4002024-10-04T22:12:06.565ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
4012024-10-04T22:12:06.565ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4022024-10-04T22:12:06.565ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
4032024-10-04T22:12:06.574ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:367
4042024-10-04T22:12:06.574ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool)
4052024-10-04T22:12:06.708ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool), kind: External }
4062024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:217
4072024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:219
4082024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:223
4092024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:225
4102024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4112024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4122024-10-04T22:12:06.716ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4132024-10-04T22:12:06.772ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4142024-10-04T22:12:06.780ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_d4135165-e105-4373-9d12-4efe0b6e4c50/crypt/zone file = sled-storage/src/dataset.rs:280
4152024-10-04T22:12:07.039ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_d4135165-e105-4373-9d12-4efe0b6e4c50 file = sled-storage/src/dataset.rs:366
4162024-10-04T22:12:07.047ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4172024-10-04T22:12:07.047ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4182024-10-04T22:12:07.047ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4192024-10-04T22:12:07.047ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4202024-10-04T22:12:07.047ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4212024-10-04T22:12:07.057ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:367
4222024-10-04T22:12:07.057ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:373 id = 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool)
4232024-10-04T22:12:07.186ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:173 zpool = ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External }
4242024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:217
4252024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:219
4262024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:223
4272024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:225
4282024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4292024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4302024-10-04T22:12:07.194ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:237
4312024-10-04T22:12:07.220ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4322024-10-04T22:12:07.262ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4332024-10-04T22:12:07.270ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987/crypt/zone file = sled-storage/src/dataset.rs:280
4342024-10-04T22:12:07.540ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 file = sled-storage/src/dataset.rs:366
4352024-10-04T22:12:07.548ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4362024-10-04T22:12:07.548ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4372024-10-04T22:12:07.548ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4382024-10-04T22:12:07.549ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: f89edb19-85d1-4199-b7bf-a91aa5ecc8e8 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, number_of_slots: 1, slot_firmware_versions: [Some("SYNTH1")] } }, zpool_name: ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4392024-10-04T22:12:07.549ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4402024-10-04T22:12:07.549ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4412024-10-04T22:12:07.549ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4422024-10-04T22:12:07.549ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:868 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4432024-10-04T22:12:07.549ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4442024-10-04T22:12:07.549ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:894 generation = Generation(1) sled_id = 544b0af3-4a18-4467-8697-af690997796e
4452024-10-04T22:12:07.549ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:899 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4462024-10-04T22:12:07.549ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:904 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4472024-10-04T22:12:07.549ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = 544b0af3-4a18-4467-8697-af690997796e
4482024-10-04T22:12:07.549ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:910 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4492024-10-04T22:12:07.549ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:757 gen = Generation(1)
4502024-10-04T22:12:07.549ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:915 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4512024-10-04T22:12:07.549ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6046346 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:55039 req_id = 41c5d769-f554-4259-9abf-c424ed96eb0d response_code = 200 uri = /omicron-physical-disks
4522024-10-04T22:12:07.580ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
4532024-10-04T22:12:07.580ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37629
4542024-10-04T22:12:07.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4552024-10-04T22:12:07.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
4562024-10-04T22:12:07.671ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4572024-10-04T22:12:07.671ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
4582024-10-04T22:12:07.739ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4592024-10-04T22:12:07.739ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4602024-10-04T22:12:07.811ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4612024-10-04T22:12:07.811ZINFOSledAgent (ServiceManager): No ledger in /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:188
4622024-10-04T22:12:07.811ZINFOSledAgent (ServiceManager): No ledger in /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:188
4632024-10-04T22:12:07.811ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4642024-10-04T22:12:07.856ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4652024-10-04T22:12:07.856ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4662024-10-04T22:12:07.856ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4672024-10-04T22:12:07.906ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3 file = illumos-utils/src/zone.rs:316
4682024-10-04T22:12:07.910ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c file = illumos-utils/src/zone.rs:316
4692024-10-04T22:12:07.914ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0 file = illumos-utils/src/zone.rs:316
4702024-10-04T22:12:07.927ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3 file = illumos-utils/src/zone.rs:354
4712024-10-04T22:12:07.932ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c file = illumos-utils/src/zone.rs:354
4722024-10-04T22:12:07.936ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0 file = illumos-utils/src/zone.rs:354
4732024-10-04T22:12:08.222ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/probe_manager.rs:230 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4742024-10-04T22:12:09.465ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4752024-10-04T22:12:09.465ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4762024-10-04T22:12:09.943ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4772024-10-04T22:12:10.420ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4782024-10-04T22:12:10.422ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4792024-10-04T22:12:10.422ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
4802024-10-04T22:12:10.424ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4812024-10-04T22:12:10.424ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
4822024-10-04T22:12:10.428ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
4832024-10-04T22:12:10.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 32.394446ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
4842024-10-04T22:12:10.556ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 27.587984ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
4852024-10-04T22:12:10.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 43.310127ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
4862024-10-04T22:12:10.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 127.007194ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
4872024-10-04T22:12:10.594ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 101.802856ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
4882024-10-04T22:12:10.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 85.74393ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
4892024-10-04T22:12:10.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 174.688823ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
4902024-10-04T22:12:10.941ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5437256286880178260 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4912024-10-04T22:12:10.943ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4634763497782449157 sled_id = 544b0af3-4a18-4467-8697-af690997796e
4922024-10-04T22:12:10.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 102.154845ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
4932024-10-04T22:12:11.010ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 209.111268ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
4942024-10-04T22:12:11.086ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 366.769453ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
4952024-10-04T22:12:11.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 254.256202ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
4962024-10-04T22:12:11.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 422.388247ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
4972024-10-04T22:12:11.247ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4269
4982024-10-04T22:12:11.247ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4279 gateway = fd00:1122:3344:101::1
4992024-10-04T22:12:11.265ZINFOSledAgent (ServiceManager): Default route already exists file = sled-agent/src/services.rs:4293 gateway = fd00:1122:3344:101::1
5002024-10-04T22:12:11.265ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4358
5012024-10-04T22:12:11.378ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 586.892496ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
5022024-10-04T22:12:11.445ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4406
5032024-10-04T22:12:11.445ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4433
5042024-10-04T22:12:11.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 765.292038ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5052024-10-04T22:12:11.490ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4446
5062024-10-04T22:12:11.490ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4310
5072024-10-04T22:12:11.538ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:43187
5082024-10-04T22:12:11.539ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:43187 req_id = d34fb3f0-2aec-4599-923c-8b1ecda1183b response_code = 200 uri = /sled-identifiers
5092024-10-04T22:12:11.545ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4352
5102024-10-04T22:12:11.545ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4413 rack_subnet = fd00:1122:3344::
5112024-10-04T22:12:11.565ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4424
5122024-10-04T22:12:11.565ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4466
5132024-10-04T22:12:11.637ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4498
5142024-10-04T22:12:11.637ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4504
5152024-10-04T22:12:11.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 472.392645ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5162024-10-04T22:12:11.730ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4545
5172024-10-04T22:12:11.730ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5182024-10-04T22:12:11.762ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/544b0af3-4a18-4467-8697-af690997796e) file = sled-agent/src/nexus.rs:571 sled_id = 544b0af3-4a18-4467-8697-af690997796e
5192024-10-04T22:12:11.763ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5202024-10-04T22:12:11.795ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5212024-10-04T22:12:11.795ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5222024-10-04T22:12:11.795ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5232024-10-04T22:12:11.795ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5242024-10-04T22:12:11.832ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
5252024-10-04T22:12:11.882ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5262024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5272024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5282024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5292024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5302024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5312024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5322024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5332024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5342024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5352024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5362024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5372024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5382024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5392024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5402024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5412024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5422024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5432024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5442024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5452024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5462024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5472024-10-04T22:12:11.882ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5482024-10-04T22:12:11.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 1.311036542s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
5492024-10-04T22:12:12.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 1.002976238s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5502024-10-04T22:12:12.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 1.192674094s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5512024-10-04T22:12:12.409ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2329353703858118291 sled_id = 544b0af3-4a18-4467-8697-af690997796e
5522024-10-04T22:12:12.409ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9526312079148586375 sled_id = 544b0af3-4a18-4467-8697-af690997796e
5532024-10-04T22:12:13.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 1.259644939s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5542024-10-04T22:12:13.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 1.107707274s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
5552024-10-04T22:12:13.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 1.158492344s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5562024-10-04T22:12:14.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 1.487872078s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
5572024-10-04T22:12:14.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 901.907668ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5582024-10-04T22:12:14.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 926.865822ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5592024-10-04T22:12:15.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 852.756328ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5602024-10-04T22:12:15.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 650.61656ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5612024-10-04T22:12:15.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0") failed: Property not found. retry in 1.307117056s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
5622024-10-04T22:12:16.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 599.344542ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5632024-10-04T22:12:16.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3") failed: Property not found. retry in 836.798473ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5642024-10-04T22:12:16.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c") failed: Property not found. retry in 697.664405ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5652024-10-04T22:12:17.532ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_173a8178-efd5-4ec1-9f4c-a05bc972d94c
5662024-10-04T22:12:17.532ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_2cceb95f-37ad-478b-be36-e32fb96c3dd0
5672024-10-04T22:12:17.532ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_internal_dns_8184f7df-5baa-40ae-9a32-5ef324ff5be3
5682024-10-04T22:12:17.532ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5692024-10-04T22:12:17.533ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5702024-10-04T22:12:17.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9952405 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:37629 req_id = e24a91e3-a168-44e2-9a3c-98b3d65f94d3 response_code = 204 uri = /omicron-zones
5712024-10-04T22:12:17.535ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5722024-10-04T22:12:17.603ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5732024-10-04T22:12:17.667ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:575
5742024-10-04T22:12:17.755ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:611
5752024-10-04T22:12:17.755ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5762024-10-04T22:12:17.787ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5772024-10-04T22:12:17.788ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5782024-10-04T22:12:17.788ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5792024-10-04T22:12:17.816ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
5802024-10-04T22:12:17.817ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38313
5812024-10-04T22:12:17.817ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5822024-10-04T22:12:17.817ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
5832024-10-04T22:12:17.837ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5842024-10-04T22:12:17.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
5852024-10-04T22:12:17.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5862024-10-04T22:12:17.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5872024-10-04T22:12:17.875ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5882024-10-04T22:12:17.875ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
5892024-10-04T22:12:17.875ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
5902024-10-04T22:12:17.875ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5912024-10-04T22:12:17.886ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5922024-10-04T22:12:17.919ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5932024-10-04T22:12:17.920ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5942024-10-04T22:12:17.920ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5952024-10-04T22:12:17.950ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 255, 32]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5962024-10-04T22:12:17.950ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = boundary_ntp
5972024-10-04T22:12:17.950ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5982024-10-04T22:12:17.950ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5992024-10-04T22:12:17.951ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
6002024-10-04T22:12:17.951ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6012024-10-04T22:12:17.974ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf file = illumos-utils/src/zone.rs:316
6022024-10-04T22:12:17.993ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf file = illumos-utils/src/zone.rs:354
6032024-10-04T22:12:18.615ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
6042024-10-04T22:12:18.615ZINFOSledAgent (ServiceManager): Booting oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6052024-10-04T22:12:18.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 38.659145ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6062024-10-04T22:12:18.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 92.954511ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6072024-10-04T22:12:18.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 247.69792ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6082024-10-04T22:12:19.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 534.976868ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6092024-10-04T22:12:19.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 893.498736ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6102024-10-04T22:12:20.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 606.834675ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6112024-10-04T22:12:20.943ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 5437256286880178260 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6122024-10-04T22:12:20.943ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 4634763497782449157 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6132024-10-04T22:12:21.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 695.191613ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6142024-10-04T22:12:21.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 576.920511ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6152024-10-04T22:12:22.430ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 2329353703858118291 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6162024-10-04T22:12:22.430ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9526312079148586375 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6172024-10-04T22:12:22.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 636.068253ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6182024-10-04T22:12:23.176ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf") failed: Property not found. retry in 878.165385ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6192024-10-04T22:12:24.075ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf
6202024-10-04T22:12:24.075ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6212024-10-04T22:12:24.075ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6222024-10-04T22:12:24.076ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6259032 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38313 req_id = 433301e3-4ef7-42af-b191-92b8771ee9fb response_code = 204 uri = /omicron-zones
6232024-10-04T22:12:24.078ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:652
6242024-10-04T22:12:24.109ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6252024-10-04T22:12:24.109ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59138
6262024-10-04T22:12:24.117ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6272024-10-04T22:12:24.117ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8543 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59138 req_id = 4fb764d4-4fbb-4021-b43f-8363e41fa329 response_code = 500 uri = /timesync
6282024-10-04T22:12:24.120ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6292024-10-04T22:12:24.253ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6302024-10-04T22:12:24.253ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:34391
6312024-10-04T22:12:28.261ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6322024-10-04T22:12:28.262ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 4008442 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:34391 req_id = 2af9c91b-02b3-43e7-8839-3dca055e448a response_code = 500 uri = /timesync
6332024-10-04T22:12:28.264ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 14536653635068036851 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6342024-10-04T22:12:28.264ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 15065028175896832703 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6352024-10-04T22:12:28.264ZERROSledAgent (kstat-sampler-worker): failed to sample kstat target, requeueing error = NoSuchKstat file = oximeter/instruments/src/kstat/sampler.rs:496 id = 9449651798875577839 sled_id = 544b0af3-4a18-4467-8697-af690997796e
6362024-10-04T22:12:28.265ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6372024-10-04T22:12:28.414ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6382024-10-04T22:12:28.414ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64302
6392024-10-04T22:12:28.423ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6402024-10-04T22:12:28.423ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 8956 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64302 req_id = 6f915b09-bdea-41d5-b7b6-3a969daa2a5d response_code = 500 uri = /timesync
6412024-10-04T22:12:28.426ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6422024-10-04T22:12:28.614ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6432024-10-04T22:12:28.614ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42090
6442024-10-04T22:12:28.623ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_de3cd5a4-cb05-4666-9b38-ab60ab962abf': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3877
6452024-10-04T22:12:28.623ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 9020 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42090 req_id = 3f98033a-f1df-47ed-ac8b-88317eca3081 response_code = 500 uri = /timesync
6462024-10-04T22:12:28.625ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6472024-10-04T22:12:28.825ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6482024-10-04T22:12:28.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:42105
6492024-10-04T22:12:28.834ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8225 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:42105 req_id = 56aba696-f6b4-422d-b7c7-69b9ce9ba83f response_code = 200 uri = /timesync
6502024-10-04T22:12:28.836ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6512024-10-04T22:12:28.836ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6522024-10-04T22:12:28.993ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6532024-10-04T22:12:28.993ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33013
6542024-10-04T22:12:29.000ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6689 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33013 req_id = 00dbbd71-1632-4e43-b23b-b17dc25c5ec0 response_code = 200 uri = /timesync
6552024-10-04T22:12:29.001ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6562024-10-04T22:12:29.001ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6572024-10-04T22:12:29.343ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6582024-10-04T22:12:29.343ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63060
6592024-10-04T22:12:29.349ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6288 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63060 req_id = 6ad23b79-a8a4-4cdd-a928-2980667d66e2 response_code = 200 uri = /timesync
6602024-10-04T22:12:29.351ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6612024-10-04T22:12:29.351ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6622024-10-04T22:12:29.738ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6632024-10-04T22:12:29.738ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54953
6642024-10-04T22:12:29.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7516 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54953 req_id = 891cd472-4765-4f2d-96f7-e9e8f4c91225 response_code = 200 uri = /timesync
6652024-10-04T22:12:29.747ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6662024-10-04T22:12:29.747ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6672024-10-04T22:12:30.010ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6682024-10-04T22:12:30.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36708
6692024-10-04T22:12:30.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7394 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36708 req_id = 56370b31-8208-40cc-b0a2-a95e62a7406e response_code = 200 uri = /timesync
6702024-10-04T22:12:30.019ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:660
6712024-10-04T22:12:30.019ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:682
6722024-10-04T22:12:30.205ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:632
6732024-10-04T22:12:30.205ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:58328
6742024-10-04T22:12:30.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 18759 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:58328 req_id = 087a9b3f-6215-483f-b523-fd495b8df192 response_code = 200 uri = /timesync
6752024-10-04T22:12:30.228ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 1241950280, ip_addr: 74.6.168.72, stratum: 3, ref_time: 1728079950.0235322, correction: -0.041443814 } file = sled-agent/src/rack_setup/service.rs:660
6762024-10-04T22:12:30.228ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1299
6772024-10-04T22:12:30.258ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
6782024-10-04T22:12:30.259ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:38324
6792024-10-04T22:12:30.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6802024-10-04T22:12:30.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
6812024-10-04T22:12:30.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6822024-10-04T22:12:30.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
6832024-10-04T22:12:30.302ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6842024-10-04T22:12:30.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6852024-10-04T22:12:30.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6862024-10-04T22:12:30.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
6872024-10-04T22:12:30.393ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6882024-10-04T22:12:30.393ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
6892024-10-04T22:12:30.451ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6902024-10-04T22:12:30.451ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6912024-10-04T22:12:30.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6922024-10-04T22:12:30.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
6932024-10-04T22:12:30.567ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6942024-10-04T22:12:30.567ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
6952024-10-04T22:12:30.636ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6962024-10-04T22:12:30.636ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
6972024-10-04T22:12:30.636ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
6982024-10-04T22:12:30.636ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6992024-10-04T22:12:30.670ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
7002024-10-04T22:12:30.670ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
7012024-10-04T22:12:30.670ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
7022024-10-04T22:12:30.670ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
7032024-10-04T22:12:30.670ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
7042024-10-04T22:12:30.670ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 411295 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 60e56fc8-fec2-4653-bae3-997c0fb7b90f response_code = 500 uri = /omicron-zones
7052024-10-04T22:12:30.670ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "60e56fc8-fec2-4653-bae3-997c0fb7b90f", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "60e56fc8-fec2-4653-bae3-997c0fb7b90f" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 128.590419ms sled_address = [fd00:1122:3344:101::1]:12345
7062024-10-04T22:12:30.799ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7072024-10-04T22:12:30.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7082024-10-04T22:12:30.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
7092024-10-04T22:12:30.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7102024-10-04T22:12:30.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7112024-10-04T22:12:30.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7122024-10-04T22:12:30.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7132024-10-04T22:12:30.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7142024-10-04T22:12:30.852ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
7152024-10-04T22:12:30.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7162024-10-04T22:12:30.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
7172024-10-04T22:12:30.885ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7182024-10-04T22:12:30.885ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7192024-10-04T22:12:30.904ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7202024-10-04T22:12:30.904ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7212024-10-04T22:12:30.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7222024-10-04T22:12:30.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
7232024-10-04T22:12:30.940ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7242024-10-04T22:12:30.940ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
7252024-10-04T22:12:30.940ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
7262024-10-04T22:12:30.940ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7272024-10-04T22:12:30.970ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
7282024-10-04T22:12:30.970ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
7292024-10-04T22:12:30.970ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
7302024-10-04T22:12:30.970ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
7312024-10-04T22:12:30.970ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
7322024-10-04T22:12:30.970ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 170663 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 8eaf6246-d0d8-4947-aacf-81f80600ee17 response_code = 500 uri = /omicron-zones
7332024-10-04T22:12:30.970ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8eaf6246-d0d8-4947-aacf-81f80600ee17", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8eaf6246-d0d8-4947-aacf-81f80600ee17" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 79.235444ms sled_address = [fd00:1122:3344:101::1]:12345
7342024-10-04T22:12:31.050ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7352024-10-04T22:12:31.051ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7362024-10-04T22:12:31.051ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
7372024-10-04T22:12:31.070ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7382024-10-04T22:12:31.070ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7392024-10-04T22:12:31.087ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7402024-10-04T22:12:31.087ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7412024-10-04T22:12:31.103ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7422024-10-04T22:12:31.103ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
7432024-10-04T22:12:31.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7442024-10-04T22:12:31.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
7452024-10-04T22:12:31.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7462024-10-04T22:12:31.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7472024-10-04T22:12:31.155ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7482024-10-04T22:12:31.155ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7492024-10-04T22:12:31.172ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7502024-10-04T22:12:31.172ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
7512024-10-04T22:12:31.190ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7522024-10-04T22:12:31.190ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
7532024-10-04T22:12:31.190ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
7542024-10-04T22:12:31.190ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7552024-10-04T22:12:31.221ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
7562024-10-04T22:12:31.221ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
7572024-10-04T22:12:31.221ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
7582024-10-04T22:12:31.221ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
7592024-10-04T22:12:31.221ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
7602024-10-04T22:12:31.221ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 170960 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 871c1c79-b51c-4a80-9826-31d2dd7c406f response_code = 500 uri = /omicron-zones
7612024-10-04T22:12:31.221ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "871c1c79-b51c-4a80-9826-31d2dd7c406f", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "871c1c79-b51c-4a80-9826-31d2dd7c406f" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 168.970594ms sled_address = [fd00:1122:3344:101::1]:12345
7622024-10-04T22:12:31.391ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7632024-10-04T22:12:31.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7642024-10-04T22:12:31.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
7652024-10-04T22:12:31.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7662024-10-04T22:12:31.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7672024-10-04T22:12:31.426ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7682024-10-04T22:12:31.426ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7692024-10-04T22:12:31.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7702024-10-04T22:12:31.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
7712024-10-04T22:12:31.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7722024-10-04T22:12:31.462ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
7732024-10-04T22:12:31.480ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7742024-10-04T22:12:31.480ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7752024-10-04T22:12:31.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7762024-10-04T22:12:31.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7772024-10-04T22:12:31.515ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7782024-10-04T22:12:31.515ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
7792024-10-04T22:12:31.534ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7802024-10-04T22:12:31.534ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
7812024-10-04T22:12:31.534ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
7822024-10-04T22:12:31.534ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7832024-10-04T22:12:31.566ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
7842024-10-04T22:12:31.566ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
7852024-10-04T22:12:31.566ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
7862024-10-04T22:12:31.566ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
7872024-10-04T22:12:31.566ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
7882024-10-04T22:12:31.566ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 175575 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 3d736e0b-527f-491e-b4dc-7293bf43521a response_code = 500 uri = /omicron-zones
7892024-10-04T22:12:31.566ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3d736e0b-527f-491e-b4dc-7293bf43521a", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3d736e0b-527f-491e-b4dc-7293bf43521a" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 168.593637ms sled_address = [fd00:1122:3344:101::1]:12345
7902024-10-04T22:12:31.735ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
7912024-10-04T22:12:31.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7922024-10-04T22:12:31.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
7932024-10-04T22:12:31.753ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7942024-10-04T22:12:31.753ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
7952024-10-04T22:12:31.771ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7962024-10-04T22:12:31.771ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7972024-10-04T22:12:31.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7982024-10-04T22:12:31.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
7992024-10-04T22:12:31.808ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8002024-10-04T22:12:31.808ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
8012024-10-04T22:12:31.826ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8022024-10-04T22:12:31.826ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8032024-10-04T22:12:31.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8042024-10-04T22:12:31.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8052024-10-04T22:12:31.862ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8062024-10-04T22:12:31.862ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
8072024-10-04T22:12:31.880ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8082024-10-04T22:12:31.880ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
8092024-10-04T22:12:31.880ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
8102024-10-04T22:12:31.880ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8112024-10-04T22:12:31.911ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
8122024-10-04T22:12:31.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
8132024-10-04T22:12:31.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
8142024-10-04T22:12:31.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
8152024-10-04T22:12:31.912ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
8162024-10-04T22:12:31.912ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 176593 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 0de94f4e-0dc5-4142-8bff-91ba10c8ada6 response_code = 500 uri = /omicron-zones
8172024-10-04T22:12:31.912ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0de94f4e-0dc5-4142-8bff-91ba10c8ada6", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:31 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0de94f4e-0dc5-4142-8bff-91ba10c8ada6" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 208.209564ms sled_address = [fd00:1122:3344:101::1]:12345
8182024-10-04T22:12:32.121ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8192024-10-04T22:12:32.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8202024-10-04T22:12:32.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
8212024-10-04T22:12:32.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8222024-10-04T22:12:32.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8232024-10-04T22:12:32.159ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8242024-10-04T22:12:32.159ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8252024-10-04T22:12:32.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8262024-10-04T22:12:32.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
8272024-10-04T22:12:32.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8282024-10-04T22:12:32.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
8292024-10-04T22:12:32.212ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8302024-10-04T22:12:32.212ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8312024-10-04T22:12:32.230ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8322024-10-04T22:12:32.230ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8332024-10-04T22:12:32.248ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8342024-10-04T22:12:32.248ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
8352024-10-04T22:12:32.266ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8362024-10-04T22:12:32.266ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
8372024-10-04T22:12:32.266ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
8382024-10-04T22:12:32.266ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8392024-10-04T22:12:32.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
8402024-10-04T22:12:32.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
8412024-10-04T22:12:32.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
8422024-10-04T22:12:32.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
8432024-10-04T22:12:32.299ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
8442024-10-04T22:12:32.299ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 178897 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 86a733de-996b-464d-a880-b99e1e762677 response_code = 500 uri = /omicron-zones
8452024-10-04T22:12:32.299ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "86a733de-996b-464d-a880-b99e1e762677", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "86a733de-996b-464d-a880-b99e1e762677" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 307.958639ms sled_address = [fd00:1122:3344:101::1]:12345
8462024-10-04T22:12:32.607ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8472024-10-04T22:12:32.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8482024-10-04T22:12:32.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
8492024-10-04T22:12:32.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8502024-10-04T22:12:32.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8512024-10-04T22:12:32.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8522024-10-04T22:12:32.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8532024-10-04T22:12:32.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8542024-10-04T22:12:32.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
8552024-10-04T22:12:32.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8562024-10-04T22:12:32.682ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
8572024-10-04T22:12:32.700ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8582024-10-04T22:12:32.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8592024-10-04T22:12:32.719ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8602024-10-04T22:12:32.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8612024-10-04T22:12:32.737ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8622024-10-04T22:12:32.737ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
8632024-10-04T22:12:32.755ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8642024-10-04T22:12:32.755ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
8652024-10-04T22:12:32.755ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
8662024-10-04T22:12:32.755ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8672024-10-04T22:12:32.787ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
8682024-10-04T22:12:32.787ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
8692024-10-04T22:12:32.787ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
8702024-10-04T22:12:32.787ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
8712024-10-04T22:12:32.787ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
8722024-10-04T22:12:32.787ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 180291 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 7420b463-c504-4ae5-9933-52805f0f48c4 response_code = 500 uri = /omicron-zones
8732024-10-04T22:12:32.787ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7420b463-c504-4ae5-9933-52805f0f48c4", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7420b463-c504-4ae5-9933-52805f0f48c4" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 357.768156ms sled_address = [fd00:1122:3344:101::1]:12345
8742024-10-04T22:12:33.145ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
8752024-10-04T22:12:33.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8762024-10-04T22:12:33.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
8772024-10-04T22:12:33.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8782024-10-04T22:12:33.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
8792024-10-04T22:12:33.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8802024-10-04T22:12:33.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8812024-10-04T22:12:33.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8822024-10-04T22:12:33.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
8832024-10-04T22:12:33.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8842024-10-04T22:12:33.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
8852024-10-04T22:12:33.239ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8862024-10-04T22:12:33.239ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8872024-10-04T22:12:33.257ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8882024-10-04T22:12:33.257ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8892024-10-04T22:12:33.275ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8902024-10-04T22:12:33.275ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
8912024-10-04T22:12:33.293ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8922024-10-04T22:12:33.293ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
8932024-10-04T22:12:33.294ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
8942024-10-04T22:12:33.294ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8952024-10-04T22:12:33.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
8962024-10-04T22:12:33.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
8972024-10-04T22:12:33.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
8982024-10-04T22:12:33.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
8992024-10-04T22:12:33.328ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9002024-10-04T22:12:33.328ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 183300 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = c36ddffc-1434-4505-93f8-c4a2dac0802c response_code = 500 uri = /omicron-zones
9012024-10-04T22:12:33.328ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c36ddffc-1434-4505-93f8-c4a2dac0802c", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c36ddffc-1434-4505-93f8-c4a2dac0802c" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 351.499131ms sled_address = [fd00:1122:3344:101::1]:12345
9022024-10-04T22:12:33.679ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9032024-10-04T22:12:33.679ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9042024-10-04T22:12:33.679ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
9052024-10-04T22:12:33.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9062024-10-04T22:12:33.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
9072024-10-04T22:12:33.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9082024-10-04T22:12:33.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9092024-10-04T22:12:33.736ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9102024-10-04T22:12:33.736ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
9112024-10-04T22:12:33.755ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9122024-10-04T22:12:33.755ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
9132024-10-04T22:12:33.773ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9142024-10-04T22:12:33.773ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9152024-10-04T22:12:33.792ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9162024-10-04T22:12:33.792ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9172024-10-04T22:12:33.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9182024-10-04T22:12:33.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
9192024-10-04T22:12:33.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9202024-10-04T22:12:33.829ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
9212024-10-04T22:12:33.829ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
9222024-10-04T22:12:33.829ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9232024-10-04T22:12:33.862ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
9242024-10-04T22:12:33.862ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9252024-10-04T22:12:33.862ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9262024-10-04T22:12:33.862ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9272024-10-04T22:12:33.862ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3292 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
9282024-10-04T22:12:33.863ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a", TimeNotSynchronized), ("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d", TimeNotSynchronized), ("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a", TimeNotSynchronized), ("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed", TimeNotSynchronized), ("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec", TimeNotSynchronized)] file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:938 latency_us = 183802 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 9fdb4757-9421-4891-869c-f121a2b5d493 response_code = 500 uri = /omicron-zones
9292024-10-04T22:12:33.863ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9fdb4757-9421-4891-869c-f121a2b5d493", "content-length": "124", "date": "Fri, 04 Oct 2024 22:12:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9fdb4757-9421-4891-869c-f121a2b5d493" } file = sled-agent/src/rack_setup/service.rs:495 retry_after = 334.05867ms sled_address = [fd00:1122:3344:101::1]:12345
9302024-10-04T22:12:34.197ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
9312024-10-04T22:12:34.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9322024-10-04T22:12:34.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
9332024-10-04T22:12:34.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9342024-10-04T22:12:34.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
9352024-10-04T22:12:34.236ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9362024-10-04T22:12:34.236ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9372024-10-04T22:12:34.254ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9382024-10-04T22:12:34.254ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
9392024-10-04T22:12:34.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9402024-10-04T22:12:34.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
9412024-10-04T22:12:34.294ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9422024-10-04T22:12:34.294ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9432024-10-04T22:12:34.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9442024-10-04T22:12:34.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9452024-10-04T22:12:34.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9462024-10-04T22:12:34.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
9472024-10-04T22:12:34.352ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9482024-10-04T22:12:34.352ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
9492024-10-04T22:12:34.352ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
9502024-10-04T22:12:34.352ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9512024-10-04T22:12:34.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9522024-10-04T22:12:34.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9532024-10-04T22:12:34.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9542024-10-04T22:12:34.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9552024-10-04T22:12:34.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9562024-10-04T22:12:34.530ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed file = illumos-utils/src/zone.rs:316
9572024-10-04T22:12:34.535ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a file = illumos-utils/src/zone.rs:316
9582024-10-04T22:12:34.539ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a file = illumos-utils/src/zone.rs:316
9592024-10-04T22:12:34.543ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec file = illumos-utils/src/zone.rs:316
9602024-10-04T22:12:34.548ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d file = illumos-utils/src/zone.rs:316
9612024-10-04T22:12:34.552ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed file = illumos-utils/src/zone.rs:354
9622024-10-04T22:12:34.557ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a file = illumos-utils/src/zone.rs:354
9632024-10-04T22:12:34.562ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a file = illumos-utils/src/zone.rs:354
9642024-10-04T22:12:34.566ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec file = illumos-utils/src/zone.rs:354
9652024-10-04T22:12:34.571ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d file = illumos-utils/src/zone.rs:354
9662024-10-04T22:12:37.017ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 165.916121ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 544b0af3-4a18-4467-8697-af690997796e
9672024-10-04T22:12:37.563ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e9c00a4e-9176-4282-bbd6-c7c66cccbfed'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9682024-10-04T22:12:37.563ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='670b5e38-d233-4bb0-b4b7-0948121b4c1a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9692024-10-04T22:12:37.563ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9702024-10-04T22:12:37.569ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9712024-10-04T22:12:37.588ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='e390af1e-afab-45e5-8d2a-d6c9468db69d'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9722024-10-04T22:12:37.588ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9732024-10-04T22:12:37.594ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='2213db84-49a0-49f2-92d5-2978058a97ec'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9742024-10-04T22:12:37.595ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
9752024-10-04T22:12:37.652ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='66b355c4-8502-43c4-b111-0ccc1116851a'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
9762024-10-04T22:12:37.652ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
9772024-10-04T22:12:37.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 34.831679ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9782024-10-04T22:12:37.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 32.166277ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9792024-10-04T22:12:37.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 34.222322ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
9802024-10-04T22:12:37.740ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 36.458829ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9812024-10-04T22:12:37.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 109.086811ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9822024-10-04T22:12:37.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 98.661281ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9832024-10-04T22:12:37.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 139.806977ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
9842024-10-04T22:12:38.019ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 216.896298ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9852024-10-04T22:12:38.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 69.979488ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
9862024-10-04T22:12:38.528ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 291.217904ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9872024-10-04T22:12:38.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 207.563679ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
9882024-10-04T22:12:38.660ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 142.80588ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9892024-10-04T22:12:38.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 376.60794ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9902024-10-04T22:12:38.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 126.814993ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
9912024-10-04T22:12:38.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 430.652019ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9922024-10-04T22:12:38.850ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 202.986208ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9932024-10-04T22:12:38.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 507.774607ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
9942024-10-04T22:12:38.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 191.193906ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
9952024-10-04T22:12:39.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 446.670235ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
9962024-10-04T22:12:39.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 564.074185ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
9972024-10-04T22:12:39.189ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 440.502601ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
9982024-10-04T22:12:39.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 1.043734185s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
9992024-10-04T22:12:39.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 496.302733ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10002024-10-04T22:12:39.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 924.407176ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10012024-10-04T22:12:39.676ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 729.193509ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10022024-10-04T22:12:39.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.146837725s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10032024-10-04T22:12:40.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 997.657816ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10042024-10-04T22:12:40.319ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 1.027073205s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10052024-10-04T22:12:40.413ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 1.127276475s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10062024-10-04T22:12:40.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 974.92667ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10072024-10-04T22:12:40.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.035684544s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10082024-10-04T22:12:41.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.176098158s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10092024-10-04T22:12:41.137ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42785
10102024-10-04T22:12:41.137ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42785 req_id = db2bb89a-dfb3-4a5a-9598-f88fdb09dfda response_code = 200 uri = /baseboard
10112024-10-04T22:12:41.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 1.063762038s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10122024-10-04T22:12:41.539ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.358862074s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10132024-10-04T22:12:41.548ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 695.637306ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10142024-10-04T22:12:41.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 718.083115ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10152024-10-04T22:12:42.256ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 1.487702757s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10162024-10-04T22:12:42.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 776.573085ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10172024-10-04T22:12:42.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 538.197656ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10182024-10-04T22:12:42.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 934.76061ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10192024-10-04T22:12:42.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.369374432s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10202024-10-04T22:12:42.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 839.494903ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10212024-10-04T22:12:43.069ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 656.128323ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10222024-10-04T22:12:43.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.020683641s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10232024-10-04T22:12:43.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 967.543424ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10242024-10-04T22:12:43.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 904.548956ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10252024-10-04T22:12:43.816ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 836.127608ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10262024-10-04T22:12:44.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 884.742126ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10272024-10-04T22:12:44.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.245778543s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10282024-10-04T22:12:44.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a") failed: Property not found. retry in 1.304364945s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10292024-10-04T22:12:44.667ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a") failed: Property not found. retry in 1.450368844s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10302024-10-04T22:12:44.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 560.127542ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10312024-10-04T22:12:45.173ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.431421134s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10322024-10-04T22:12:45.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.062861986s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10332024-10-04T22:12:45.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 943.910538ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10342024-10-04T22:12:46.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 940.721071ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10352024-10-04T22:12:46.611ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.465379748s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10362024-10-04T22:12:46.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 861.545089ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10372024-10-04T22:12:47.293ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 906.165082ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10382024-10-04T22:12:47.704ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.298167752s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10392024-10-04T22:12:48.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.150430036s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10402024-10-04T22:12:48.212ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.122135693s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10412024-10-04T22:12:49.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 826.300829ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10422024-10-04T22:12:49.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.089041372s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10432024-10-04T22:12:49.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.042945314s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10442024-10-04T22:12:49.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.374037031s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10452024-10-04T22:12:50.373ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 842.681408ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10462024-10-04T22:12:50.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.279561377s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10472024-10-04T22:12:51.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 783.709847ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10482024-10-04T22:12:51.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.118124618s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10492024-10-04T22:12:51.689ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.388938848s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10502024-10-04T22:12:52.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 835.995944ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10512024-10-04T22:12:52.192ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 274.15902ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 544b0af3-4a18-4467-8697-af690997796e
10522024-10-04T22:12:52.366ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 886.870557ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10532024-10-04T22:12:52.867ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 691.235843ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10542024-10-04T22:12:53.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 915.374431ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10552024-10-04T22:12:53.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.150782002s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10562024-10-04T22:12:53.569ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 926.093957ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10572024-10-04T22:12:54.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 529.8763ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10582024-10-04T22:12:54.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 780.465162ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10592024-10-04T22:12:54.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 990.731139ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10602024-10-04T22:12:54.552ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 926.165813ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10612024-10-04T22:12:55.213ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed") failed: Property not found. retry in 1.468846963s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10622024-10-04T22:12:55.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec") failed: Property not found. retry in 1.035410176s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10632024-10-04T22:12:55.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d") failed: Property not found. retry in 1.254417656s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10642024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_e9c00a4e-9176-4282-bbd6-c7c66cccbfed
10652024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_66b355c4-8502-43c4-b111-0ccc1116851a
10662024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_670b5e38-d233-4bb0-b4b7-0948121b4c1a
10672024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_2213db84-49a0-49f2-92d5-2978058a97ec
10682024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_cockroachdb_e390af1e-afab-45e5-8d2a-d6c9468db69d
10692024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
10702024-10-04T22:12:56.779ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
10712024-10-04T22:12:56.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22617381 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:38324 req_id = 45bbc6ba-e681-4479-b00f-362bb2b5ebd7 response_code = 204 uri = /omicron-zones
10722024-10-04T22:12:56.812ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37296
10732024-10-04T22:12:56.842ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::3]:32221 file = sled-agent/src/services.rs:3720
10742024-10-04T22:13:04.119ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3738
10752024-10-04T22:13:19.595ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3761
10762024-10-04T22:13:19.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 22784098 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:37296 req_id = 877fe099-ccb8-4305-bd87-c8177864894d response_code = 204 uri = /cockroachdb
10772024-10-04T22:13:19.599ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:56576
10782024-10-04T22:13:19.599ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 217 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:56576 req_id = 42fcf369-e1f6-48bc-8a7b-715be8ac5491 response_code = 200 uri = /baseboard
10792024-10-04T22:13:19.604ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 583.674144ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 544b0af3-4a18-4467-8697-af690997796e
10802024-10-04T22:13:19.637ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3 (service), underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0 (service), underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c (service), underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf (service), underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: fe659be2-daad-487d-9095-4acfcf2c3642, kind: Service { id: de3cd5a4-cb05-4666-9b38-ab60ab962abf }, name: Name("ntp-de3cd5a4-cb05-4666-9b38-ab60ab962abf"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 32])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 2213db84-49a0-49f2-92d5-2978058a97ec (service), underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 66b355c4-8502-43c4-b111-0ccc1116851a (service), underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed (service), underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: e390af1e-afab-45e5-8d2a-d6c9468db69d (service), underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a (service), underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }, OmicronZoneConfig { id: fcb39657-4027-45f8-9b5b-fc09e764f468 (service), underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: bbfb69f6-617d-4782-a1f5-f7e225f6c330, kind: Service { id: fcb39657-4027-45f8-9b5b-fc09e764f468 }, name: Name("external-dns-fcb39657-4027-45f8-9b5b-fc09e764f468"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 178])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: e5972f58-0215-4789-9de6-8c745fc8691c (service), underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 46b99b09-11d4-4fe4-bf41-6ed61ba36d44, kind: Service { id: e5972f58-0215-4789-9de6-8c745fc8691c }, name: Name("external-dns-e5972f58-0215-4789-9de6-8c745fc8691c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 244])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 5b35dee7-890d-4eb6-8ee1-7360a15c561a (service), underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 948c7b04-5bd4-4111-a486-7fc541ba08f2, kind: Service { id: 5b35dee7-890d-4eb6-8ee1-7360a15c561a }, name: Name("nexus-5b35dee7-890d-4eb6-8ee1-7360a15c561a"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 148, 10])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: e72dd7c9-9d72-4f49-b48b-231a411d26a3 (service), underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 13e67383-e8ea-42d3-b048-d94c625ebeb6, kind: Service { id: e72dd7c9-9d72-4f49-b48b-231a411d26a3 }, name: Name("nexus-e72dd7c9-9d72-4f49-b48b-231a411d26a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 146])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 46d21628-a828-4b09-b280-d1af35f56206 (service), underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 9195d467-8fcf-4fcc-9e20-edc728f0400f, kind: Service { id: 46d21628-a828-4b09-b280-d1af35f56206 }, name: Name("nexus-46d21628-a828-4b09-b280-d1af35f56206"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 75036a45-c39d-44e7-9208-1799441e9ebf (service), underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: ba3eccf6-4897-41d8-a079-7fc999427052 (service), underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 6436d814-77d3-42d3-9e54-70a16d185c32 (service), underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 98ea9840-3ff8-4efd-aa4d-cfad1addd77d (service), underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 67428935-72b2-44d0-8f19-d288da37d55b (service), underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 66a90fa6-7dcc-4ced-93d6-67a7ab8c3856 (service), underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External } } } }, OmicronZoneConfig { id: 230eb49b-a0c6-474e-aace-ba30ff793520 (service), underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External } } } }, OmicronZoneConfig { id: 39899e1a-3f0a-429a-ae05-77f3247d3e26 (service), underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External } } } }, OmicronZoneConfig { id: 2367eac2-ed23-44dc-8c16-a31ddf5412c7 (service), underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External } } } }, OmicronZoneConfig { id: f1bebc8f-e709-4c9a-a00c-cd503c04c45b (service), underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External } } } }, OmicronZoneConfig { id: b6678fd5-9eca-4aa7-9e47-23ad27c73c85 (service), underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External } } } }, OmicronZoneConfig { id: 72974fd7-cc49-40bb-8097-19b6d7d747d3 (service), underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool), kind: External } } } }, OmicronZoneConfig { id: 06c64b28-3935-456b-b9a9-8035576b6e7a (service), underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool), kind: External } } } }, OmicronZoneConfig { id: e74049ca-0d69-456c-8680-673308a2e271 (service), underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:458 sled_address = [fd00:1122:3344:101::1]:12345
10812024-10-04T22:13:19.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:53624
10822024-10-04T22:13:19.639ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10832024-10-04T22:13:19.639ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 8184f7df-5baa-40ae-9a32-5ef324ff5be3, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: InternalDns }, responder: ... }
10842024-10-04T22:13:19.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10852024-10-04T22:13:19.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2cceb95f-37ad-478b-be36-e32fb96c3dd0, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: InternalDns }, responder: ... }
10862024-10-04T22:13:19.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10872024-10-04T22:13:19.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 173a8178-efd5-4ec1-9f4c-a05bc972d94c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10882024-10-04T22:13:19.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10892024-10-04T22:13:19.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2213db84-49a0-49f2-92d5-2978058a97ec, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Cockroach }, responder: ... }
10902024-10-04T22:13:19.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10912024-10-04T22:13:19.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66b355c4-8502-43c4-b111-0ccc1116851a, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Cockroach }, responder: ... }
10922024-10-04T22:13:19.750ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10932024-10-04T22:13:19.750ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e9c00a4e-9176-4282-bbd6-c7c66cccbfed, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10942024-10-04T22:13:19.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10952024-10-04T22:13:19.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e390af1e-afab-45e5-8d2a-d6c9468db69d, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10962024-10-04T22:13:19.790ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10972024-10-04T22:13:19.790ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 670b5e38-d233-4bb0-b4b7-0948121b4c1a, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Cockroach }, responder: ... }
10982024-10-04T22:13:19.810ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: fcb39657-4027-45f8-9b5b-fc09e764f468, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10992024-10-04T22:13:19.810ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: fcb39657-4027-45f8-9b5b-fc09e764f468, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11002024-10-04T22:13:19.889ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e5972f58-0215-4789-9de6-8c745fc8691c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11012024-10-04T22:13:19.889ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e5972f58-0215-4789-9de6-8c745fc8691c, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11022024-10-04T22:13:19.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ba3eccf6-4897-41d8-a079-7fc999427052, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
11032024-10-04T22:13:19.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: ba3eccf6-4897-41d8-a079-7fc999427052, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11042024-10-04T22:13:20.034ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 66a90fa6-7dcc-4ced-93d6-67a7ab8c3856, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11052024-10-04T22:13:20.034ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 66a90fa6-7dcc-4ced-93d6-67a7ab8c3856, dataset_name: DatasetName { pool_name: ZpoolName { id: 676a3716-3d3e-465b-9699-cf98062dabca (zpool), kind: External }, kind: Crucible }, responder: ... }
11062024-10-04T22:13:20.105ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 230eb49b-a0c6-474e-aace-ba30ff793520, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11072024-10-04T22:13:20.105ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 230eb49b-a0c6-474e-aace-ba30ff793520, dataset_name: DatasetName { pool_name: ZpoolName { id: 9af95e51-a064-462a-bea1-8d3042c749de (zpool), kind: External }, kind: Crucible }, responder: ... }
11082024-10-04T22:13:20.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 39899e1a-3f0a-429a-ae05-77f3247d3e26, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11092024-10-04T22:13:20.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 39899e1a-3f0a-429a-ae05-77f3247d3e26, dataset_name: DatasetName { pool_name: ZpoolName { id: 9e3d11a1-af30-407b-b05f-d5f1a5762d41 (zpool), kind: External }, kind: Crucible }, responder: ... }
11102024-10-04T22:13:20.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 2367eac2-ed23-44dc-8c16-a31ddf5412c7, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11112024-10-04T22:13:20.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 2367eac2-ed23-44dc-8c16-a31ddf5412c7, dataset_name: DatasetName { pool_name: ZpoolName { id: ac4d5d54-7085-42e4-a0d6-359bebb3c91f (zpool), kind: External }, kind: Crucible }, responder: ... }
11122024-10-04T22:13:20.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f1bebc8f-e709-4c9a-a00c-cd503c04c45b, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11132024-10-04T22:13:20.322ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: f1bebc8f-e709-4c9a-a00c-cd503c04c45b, dataset_name: DatasetName { pool_name: ZpoolName { id: c4a31058-0731-4581-8d9d-c5c4b66770da (zpool), kind: External }, kind: Crucible }, responder: ... }
11142024-10-04T22:13:20.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b6678fd5-9eca-4aa7-9e47-23ad27c73c85, dataset_name: DatasetName { pool_name: ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11152024-10-04T22:13:20.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: b6678fd5-9eca-4aa7-9e47-23ad27c73c85, dataset_name: DatasetName { pool_name: ZpoolName { id: 75662bd1-8970-44ab-88a5-02e715476179 (zpool), kind: External }, kind: Crucible }, responder: ... }
11162024-10-04T22:13:20.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 72974fd7-cc49-40bb-8097-19b6d7d747d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11172024-10-04T22:13:20.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 72974fd7-cc49-40bb-8097-19b6d7d747d3, dataset_name: DatasetName { pool_name: ZpoolName { id: 1279c2fc-c95f-45e2-a67c-783305b4023a (zpool), kind: External }, kind: Crucible }, responder: ... }
11182024-10-04T22:13:20.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 06c64b28-3935-456b-b9a9-8035576b6e7a, dataset_name: DatasetName { pool_name: ZpoolName { id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11192024-10-04T22:13:20.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: 06c64b28-3935-456b-b9a9-8035576b6e7a, dataset_name: DatasetName { pool_name: ZpoolName { id: d4135165-e105-4373-9d12-4efe0b6e4c50 (zpool), kind: External }, kind: Crucible }, responder: ... }
11202024-10-04T22:13:20.598ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: e74049ca-0d69-456c-8680-673308a2e271, dataset_name: DatasetName { pool_name: ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
11212024-10-04T22:13:20.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1059 request = NewFilesystemRequest { dataset_id: e74049ca-0d69-456c-8680-673308a2e271, dataset_name: DatasetName { pool_name: ZpoolName { id: 2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987 (zpool), kind: External }, kind: Crucible }, responder: ... }
11222024-10-04T22:13:20.662ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11232024-10-04T22:13:20.662ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
11242024-10-04T22:13:20.662ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
11252024-10-04T22:13:20.662ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11262024-10-04T22:13:20.783ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
11272024-10-04T22:13:20.784ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11282024-10-04T22:13:20.785ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
11292024-10-04T22:13:20.845ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
11302024-10-04T22:13:20.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11312024-10-04T22:13:20.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11322024-10-04T22:13:20.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11332024-10-04T22:13:20.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11342024-10-04T22:13:20.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11352024-10-04T22:13:20.845ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11362024-10-04T22:13:20.856ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
11372024-10-04T22:13:20.998ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
11382024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11392024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11402024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11412024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11422024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11432024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11442024-10-04T22:13:21.059ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11452024-10-04T22:13:21.059ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32 file = illumos-utils/src/zone.rs:316
11462024-10-04T22:13:21.066ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052 file = illumos-utils/src/zone.rs:316
11472024-10-04T22:13:21.073ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b file = illumos-utils/src/zone.rs:316
11482024-10-04T22:13:21.079ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520 file = illumos-utils/src/zone.rs:316
11492024-10-04T22:13:21.084ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26 file = illumos-utils/src/zone.rs:316
11502024-10-04T22:13:21.089ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7 file = illumos-utils/src/zone.rs:316
11512024-10-04T22:13:21.094ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b file = illumos-utils/src/zone.rs:316
11522024-10-04T22:13:21.243ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32 file = illumos-utils/src/zone.rs:354
11532024-10-04T22:13:21.249ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052 file = illumos-utils/src/zone.rs:354
11542024-10-04T22:13:21.254ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b file = illumos-utils/src/zone.rs:354
11552024-10-04T22:13:21.259ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520 file = illumos-utils/src/zone.rs:354
11562024-10-04T22:13:21.264ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26 file = illumos-utils/src/zone.rs:354
11572024-10-04T22:13:21.268ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7 file = illumos-utils/src/zone.rs:354
11582024-10-04T22:13:21.273ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b file = illumos-utils/src/zone.rs:354
11592024-10-04T22:13:21.278ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf file = illumos-utils/src/zone.rs:316
11602024-10-04T22:13:21.283ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d file = illumos-utils/src/zone.rs:316
11612024-10-04T22:13:21.288ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856 file = illumos-utils/src/zone.rs:316
11622024-10-04T22:13:21.294ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85 file = illumos-utils/src/zone.rs:316
11632024-10-04T22:13:21.300ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3 file = illumos-utils/src/zone.rs:316
11642024-10-04T22:13:21.305ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a file = illumos-utils/src/zone.rs:316
11652024-10-04T22:13:21.312ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271 file = illumos-utils/src/zone.rs:316
11662024-10-04T22:13:21.364ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
11672024-10-04T22:13:21.364ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf file = illumos-utils/src/zone.rs:354
11682024-10-04T22:13:21.372ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d file = illumos-utils/src/zone.rs:354
11692024-10-04T22:13:21.380ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856 file = illumos-utils/src/zone.rs:354
11702024-10-04T22:13:21.387ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85 file = illumos-utils/src/zone.rs:354
11712024-10-04T22:13:21.394ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3 file = illumos-utils/src/zone.rs:354
11722024-10-04T22:13:21.401ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a file = illumos-utils/src/zone.rs:354
11732024-10-04T22:13:21.409ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271 file = illumos-utils/src/zone.rs:354
11742024-10-04T22:13:21.527ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
11752024-10-04T22:13:21.621ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
11762024-10-04T22:13:21.683ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
11772024-10-04T22:13:21.746ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
11782024-10-04T22:13:21.750ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11792024-10-04T22:13:21.750ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11802024-10-04T22:13:21.807ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 158, 244]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11812024-10-04T22:13:21.807ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
11822024-10-04T22:13:21.808ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11832024-10-04T22:13:21.808ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11842024-10-04T22:13:21.863ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 148, 10]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11852024-10-04T22:13:21.863ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
11862024-10-04T22:13:21.864ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11872024-10-04T22:13:21.864ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11882024-10-04T22:13:21.926ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 203, 186]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11892024-10-04T22:13:21.926ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
11902024-10-04T22:13:21.928ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11912024-10-04T22:13:21.928ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11922024-10-04T22:13:21.989ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 137, 178]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
11932024-10-04T22:13:21.989ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = external_dns
11942024-10-04T22:13:21.992ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
11952024-10-04T22:13:21.992ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
11962024-10-04T22:13:22.048ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 174, 146]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
11972024-10-04T22:13:22.048ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1266 zone_type = nexus
11982024-10-04T22:13:22.048ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
11992024-10-04T22:13:22.048ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
12002024-10-04T22:13:22.049ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
12012024-10-04T22:13:22.049ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
12022024-10-04T22:13:22.049ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
12032024-10-04T22:13:22.049ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
12042024-10-04T22:13:22.049ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
12052024-10-04T22:13:22.049ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
12062024-10-04T22:13:22.052ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
12072024-10-04T22:13:22.054ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12082024-10-04T22:13:22.100ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
12092024-10-04T22:13:22.103ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12102024-10-04T22:13:22.103ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
12112024-10-04T22:13:22.107ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12122024-10-04T22:13:22.107ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
12132024-10-04T22:13:22.110ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
12142024-10-04T22:13:22.110ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
12152024-10-04T22:13:22.110ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12162024-10-04T22:13:22.253ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
12172024-10-04T22:13:22.258ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12182024-10-04T22:13:22.258ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c file = illumos-utils/src/zone.rs:316
12192024-10-04T22:13:22.280ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a file = illumos-utils/src/zone.rs:316
12202024-10-04T22:13:22.301ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206 file = illumos-utils/src/zone.rs:316
12212024-10-04T22:13:22.337ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468 file = illumos-utils/src/zone.rs:316
12222024-10-04T22:13:22.442ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c file = illumos-utils/src/zone.rs:354
12232024-10-04T22:13:22.471ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a file = illumos-utils/src/zone.rs:354
12242024-10-04T22:13:22.501ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206 file = illumos-utils/src/zone.rs:354
12252024-10-04T22:13:22.520ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468 file = illumos-utils/src/zone.rs:354
12262024-10-04T22:13:22.543ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3 file = illumos-utils/src/zone.rs:316
12272024-10-04T22:13:22.603ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3 file = illumos-utils/src/zone.rs:354
12282024-10-04T22:13:24.579ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12292024-10-04T22:13:24.592ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12302024-10-04T22:13:24.592ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
12312024-10-04T22:13:24.596ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
12322024-10-04T22:13:24.652ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_c4a31058-0731-4581-8d9d-c5c4b66770da/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='97e29040-ba5a-44d9-bef6-d3c1672a4781'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12332024-10-04T22:13:24.653ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_1279c2fc-c95f-45e2-a67c-783305b4023a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5f9c5267-b11e-482c-ab51-cfd881f3c044'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12342024-10-04T22:13:24.653ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
12352024-10-04T22:13:24.663ZINFOSledAgent (ServiceManager): Booting oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
12362024-10-04T22:13:24.672ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_d4135165-e105-4373-9d12-4efe0b6e4c50/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='0f44c516-2aac-43d8-92d1-eb2298ef640d'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12372024-10-04T22:13:24.673ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='cf8dfb6f-328e-4477-b47e-fc910261e8ea'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12382024-10-04T22:13:24.673ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_9af95e51-a064-462a-bea1-8d3042c749de/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='62da9ab4-29f1-4a24-8caf-bbe07968f395'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12392024-10-04T22:13:24.673ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_9e3d11a1-af30-407b-b05f-d5f1a5762d41/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='a94246df-8085-45c5-901c-8b4b090d131a'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12402024-10-04T22:13:24.674ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_ac4d5d54-7085-42e4-a0d6-359bebb3c91f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5ac06d2b-25b8-41a1-8f9d-2ab7745c3bd5'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12412024-10-04T22:13:24.676ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_75662bd1-8970-44ab-88a5-02e715476179/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5f6c02cd-8fdc-42df-be2d-03cd790dd245'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12422024-10-04T22:13:24.676ZINFOSledAgent (ServiceManager): Booting oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
12432024-10-04T22:13:24.685ZINFOSledAgent (ServiceManager): Booting oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
12442024-10-04T22:13:24.697ZINFOSledAgent (ServiceManager): Booting oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
12452024-10-04T22:13:24.714ZINFOSledAgent (ServiceManager): Booting oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
12462024-10-04T22:13:24.719ZINFOSledAgent (ServiceManager): Booting oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
12472024-10-04T22:13:24.724ZINFOSledAgent (ServiceManager): Booting oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
12482024-10-04T22:13:24.874ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='75036a45-c39d-44e7-9208-1799441e9ebf'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12492024-10-04T22:13:24.875ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
12502024-10-04T22:13:25.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 25.652662ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
12512024-10-04T22:13:25.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 52.587606ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
12522024-10-04T22:13:25.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 68.2515ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
12532024-10-04T22:13:25.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 59.493839ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
12542024-10-04T22:13:25.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 48.911365ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
12552024-10-04T22:13:25.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 53.042381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
12562024-10-04T22:13:25.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 104.56075ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
12572024-10-04T22:13:25.388ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 36.852724ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
12582024-10-04T22:13:25.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 37.783695ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
12592024-10-04T22:13:25.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 119.627834ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
12602024-10-04T22:13:25.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 104.418634ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
12612024-10-04T22:13:26.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 123.201704ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
12622024-10-04T22:13:26.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 117.548647ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
12632024-10-04T22:13:26.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 44.626338ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
12642024-10-04T22:13:26.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 44.590544ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
12652024-10-04T22:13:26.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 56.869935ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
12662024-10-04T22:13:26.516ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 279.073976ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
12672024-10-04T22:13:26.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 79.150127ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
12682024-10-04T22:13:26.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 61.655076ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
12692024-10-04T22:13:26.601ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 247.024254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
12702024-10-04T22:13:26.629ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 136.501775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
12712024-10-04T22:13:26.651ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 71.318043ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
12722024-10-04T22:13:26.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 244.27125ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
12732024-10-04T22:13:26.685ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 291.391184ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
12742024-10-04T22:13:26.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 107.426592ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
12752024-10-04T22:13:26.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 55.746749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
12762024-10-04T22:13:26.746ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 203.589228ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
12772024-10-04T22:13:26.763ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 190.267504ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
12782024-10-04T22:13:26.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 158.746947ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
12792024-10-04T22:13:26.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 361.351637ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
12802024-10-04T22:13:26.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 142.170076ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
12812024-10-04T22:13:26.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 236.555721ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
12822024-10-04T22:13:26.831ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 544.720351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
12832024-10-04T22:13:26.991ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12842024-10-04T22:13:27.004ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 487.934123ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
12852024-10-04T22:13:27.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 417.277805ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
12862024-10-04T22:13:27.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 243.858477ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
12872024-10-04T22:13:27.041ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 273.448763ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
12882024-10-04T22:13:27.052ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 323.388395ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
12892024-10-04T22:13:27.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 453.89311ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
12902024-10-04T22:13:27.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 172.998351ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
12912024-10-04T22:13:27.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 416.548355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
12922024-10-04T22:13:27.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 453.590159ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
12932024-10-04T22:13:27.095ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
12942024-10-04T22:13:27.100ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_676a3716-3d3e-465b-9699-cf98062dabca/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='95e67b7a-3590-4b2f-af34-bfcdb8cb81dd'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
12952024-10-04T22:13:27.100ZINFOSledAgent (ServiceManager): Booting oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
12962024-10-04T22:13:27.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 624.154399ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
12972024-10-04T22:13:27.259ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 452.95962ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
12982024-10-04T22:13:27.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.085060736s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
12992024-10-04T22:13:27.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 455.471581ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13002024-10-04T22:13:27.346ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 41.918339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13012024-10-04T22:13:27.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 63.452256ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13022024-10-04T22:13:27.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 939.525237ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13032024-10-04T22:13:27.400ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 101.88922ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13042024-10-04T22:13:27.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 595.544566ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13052024-10-04T22:13:27.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 70.939582ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13062024-10-04T22:13:27.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 677.340472ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13072024-10-04T22:13:27.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 505.413233ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13082024-10-04T22:13:27.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 240.951004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13092024-10-04T22:13:27.529ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 244.306607ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13102024-10-04T22:13:27.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 543.628341ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13112024-10-04T22:13:27.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.049371346s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
13122024-10-04T22:13:27.565ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 848.077647ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13132024-10-04T22:13:27.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 616.525557ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13142024-10-04T22:13:27.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 577.005653ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13152024-10-04T22:13:27.793ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 575.291802ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13162024-10-04T22:13:27.804ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.222029063s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
13172024-10-04T22:13:27.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.094091613s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13182024-10-04T22:13:28.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 763.036548ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13192024-10-04T22:13:28.037ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.467271185s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13202024-10-04T22:13:28.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.282983401s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13212024-10-04T22:13:28.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.485595446s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13222024-10-04T22:13:28.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 653.704312ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13232024-10-04T22:13:28.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.18478136s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13242024-10-04T22:13:28.367ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.017859338s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13252024-10-04T22:13:28.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.010780401s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13262024-10-04T22:13:28.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.074916949s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
13272024-10-04T22:13:28.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.287382957s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13282024-10-04T22:13:28.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.113550606s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
13292024-10-04T22:13:28.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 918.256122ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13302024-10-04T22:13:28.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 909.019254ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13312024-10-04T22:13:29.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 534.924144ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13322024-10-04T22:13:29.039ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.263815406s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
13332024-10-04T22:13:29.392ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.384446545s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13342024-10-04T22:13:29.404ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.308131046s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13352024-10-04T22:13:29.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 893.672064ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13362024-10-04T22:13:29.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 623.475753ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
13372024-10-04T22:13:29.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 769.460371ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13382024-10-04T22:13:29.550ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.183898199s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13392024-10-04T22:13:29.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 768.961101ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13402024-10-04T22:13:29.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 923.965832ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13412024-10-04T22:13:29.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.415401552s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13422024-10-04T22:13:29.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 628.91463ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13432024-10-04T22:13:29.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.077110905s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13442024-10-04T22:13:29.961ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.030567596s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
13452024-10-04T22:13:30.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 557.538361ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
13462024-10-04T22:13:30.440ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.321846849s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13472024-10-04T22:13:30.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.440695243s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
13482024-10-04T22:13:30.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.015940505s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13492024-10-04T22:13:30.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.309917612s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13502024-10-04T22:13:30.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.269565878s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13512024-10-04T22:13:30.785ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 892.877388ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13522024-10-04T22:13:30.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 681.717013ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13532024-10-04T22:13:30.807ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 599.518931ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13542024-10-04T22:13:30.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 674.805446ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13552024-10-04T22:13:30.864ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.043747104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
13562024-10-04T22:13:31.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.45202733s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
13572024-10-04T22:13:31.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.304316471s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13582024-10-04T22:13:31.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 788.318629ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13592024-10-04T22:13:31.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 726.012547ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13602024-10-04T22:13:31.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 656.175767ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13612024-10-04T22:13:31.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 634.219924ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13622024-10-04T22:13:31.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.008453949s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13632024-10-04T22:13:31.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 660.959196ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13642024-10-04T22:13:31.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 570.07899ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13652024-10-04T22:13:31.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.237673321s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
13662024-10-04T22:13:31.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 968.114603ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
13672024-10-04T22:13:32.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.047101518s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13682024-10-04T22:13:32.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 946.757649ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13692024-10-04T22:13:32.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 611.757943ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13702024-10-04T22:13:32.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.308536586s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13712024-10-04T22:13:32.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.485768521s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
13722024-10-04T22:13:32.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.218052886s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13732024-10-04T22:13:32.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.365484444s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13742024-10-04T22:13:32.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 884.421391ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13752024-10-04T22:13:32.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.138637035s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13762024-10-04T22:13:32.377ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
13772024-10-04T22:13:32.378ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
13782024-10-04T22:13:32.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 509.052851ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
13792024-10-04T22:13:32.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 27.766033ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
13802024-10-04T22:13:32.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.277968619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
13812024-10-04T22:13:32.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 136.498375ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
13822024-10-04T22:13:32.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.055075843s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13832024-10-04T22:13:32.827ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 142.970502ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
13842024-10-04T22:13:32.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 664.626809ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
13852024-10-04T22:13:32.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 215.95472ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
13862024-10-04T22:13:32.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.219086585s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
13872024-10-04T22:13:33.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 549.174473ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13882024-10-04T22:13:33.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 853.780335ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
13892024-10-04T22:13:33.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.14900067s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
13902024-10-04T22:13:33.205ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.074225791s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
13912024-10-04T22:13:33.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.133691151s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
13922024-10-04T22:13:33.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.144941909s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
13932024-10-04T22:13:33.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 683.85314ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
13942024-10-04T22:13:33.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 609.612978ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
13952024-10-04T22:13:33.571ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 599.117854ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
13962024-10-04T22:13:33.595ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.097894346s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
13972024-10-04T22:13:33.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 737.204572ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
13982024-10-04T22:13:33.756ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 903.231546ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
13992024-10-04T22:13:33.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 631.08701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14002024-10-04T22:13:33.927ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 985.32905ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
14012024-10-04T22:13:33.944ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.213034993s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
14022024-10-04T22:13:34.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.453310156s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
14032024-10-04T22:13:34.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 510.609468ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
14042024-10-04T22:13:34.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.360084749s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
14052024-10-04T22:13:34.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 556.232799ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
14062024-10-04T22:13:34.288ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 502.792058ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
14072024-10-04T22:13:34.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 977.087763ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
14082024-10-04T22:13:34.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.216184466s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
14092024-10-04T22:13:34.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 600.447439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14102024-10-04T22:13:34.464ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 504.892891ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
14112024-10-04T22:13:34.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.325346985s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
14122024-10-04T22:13:34.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.258416351s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
14132024-10-04T22:13:34.698ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 862.666232ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
14142024-10-04T22:13:34.708ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.073394877s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
14152024-10-04T22:13:34.782ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 992.890486ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
14162024-10-04T22:13:34.800ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 888.145442ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
14172024-10-04T22:13:34.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.074265138s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
14182024-10-04T22:13:34.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.346624583s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
14192024-10-04T22:13:35.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 967.734998ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14202024-10-04T22:13:35.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.156304081s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
14212024-10-04T22:13:35.285ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.387208156s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 544b0af3-4a18-4467-8697-af690997796e
14222024-10-04T22:13:35.523ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 706.456258ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
14232024-10-04T22:13:35.642ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 580.762481ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
14242024-10-04T22:13:35.668ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 646.697637ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
14252024-10-04T22:13:35.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.075736762s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
14262024-10-04T22:13:35.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.191073284s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
14272024-10-04T22:13:35.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 712.355019ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
14282024-10-04T22:13:35.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 845.296749ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
14292024-10-04T22:13:35.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 657.301943ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
14302024-10-04T22:13:35.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.466912245s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
14312024-10-04T22:13:35.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 605.628468ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
14322024-10-04T22:13:35.995ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 885.064843ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14332024-10-04T22:13:36.007ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.116304946s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
14342024-10-04T22:13:36.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.131066806s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
14352024-10-04T22:13:36.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 775.343483ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
14362024-10-04T22:13:36.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.088499564s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
14372024-10-04T22:13:36.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.073600523s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
14382024-10-04T22:13:36.341ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.334166506s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
14392024-10-04T22:13:36.498ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 721.353205ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
14402024-10-04T22:13:36.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 815.996516ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
14412024-10-04T22:13:36.586ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.434435578s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
14422024-10-04T22:13:36.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 602.442072ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
14432024-10-04T22:13:36.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.00120595s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
14442024-10-04T22:13:36.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.10441437s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14452024-10-04T22:13:36.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 520.392892ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
14462024-10-04T22:13:37.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 647.572178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
14472024-10-04T22:13:37.132ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.220127795s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
14482024-10-04T22:13:37.228ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.243390958s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
14492024-10-04T22:13:37.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.214324669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
14502024-10-04T22:13:37.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 998.117355ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
14512024-10-04T22:13:37.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.015064714s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
14522024-10-04T22:13:37.420ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 888.987039ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
14532024-10-04T22:13:37.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 903.593445ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
14542024-10-04T22:13:37.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.369776139s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
14552024-10-04T22:13:37.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 958.02045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
14562024-10-04T22:13:37.734ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.202369444s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
14572024-10-04T22:13:37.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 890.937415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
14582024-10-04T22:13:37.744ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14592024-10-04T22:13:37.744ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14602024-10-04T22:13:37.744ZINFOSledAgent (ServiceManager): Booting oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
14612024-10-04T22:13:37.749ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
14622024-10-04T22:13:37.749ZINFOSledAgent (ServiceManager): Booting oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
14632024-10-04T22:13:37.754ZINFOSledAgent (ServiceManager): Booting oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
14642024-10-04T22:13:37.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.437892765s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
14652024-10-04T22:13:38.006ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.132756567s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14662024-10-04T22:13:38.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.214453529s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
14672024-10-04T22:13:38.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 51.595898ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
14682024-10-04T22:13:38.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 39.617661ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
14692024-10-04T22:13:38.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 42.767723ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
14702024-10-04T22:13:38.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 116.733768ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
14712024-10-04T22:13:38.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 62.269153ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
14722024-10-04T22:13:38.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 106.968396ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
14732024-10-04T22:13:38.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 230.203428ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
14742024-10-04T22:13:38.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 293.718913ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
14752024-10-04T22:13:38.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 192.451977ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
14762024-10-04T22:13:38.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 853.103218ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
14772024-10-04T22:13:38.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.020218872s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
14782024-10-04T22:13:38.351ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 777.579569ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
14792024-10-04T22:13:38.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 781.346273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
14802024-10-04T22:13:38.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.364426837s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
14812024-10-04T22:13:38.424ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 742.371206ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
14822024-10-04T22:13:38.453ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 344.439161ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
14832024-10-04T22:13:38.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 263.987454ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
14842024-10-04T22:13:38.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.259410984s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
14852024-10-04T22:13:38.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.168617556s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
14862024-10-04T22:13:38.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 531.661648ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
14872024-10-04T22:13:38.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.144169216s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
14882024-10-04T22:13:38.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 1.02946407s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
14892024-10-04T22:13:38.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 799.745043ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
14902024-10-04T22:13:38.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.260079359s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
14912024-10-04T22:13:38.945ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 651.831652ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
14922024-10-04T22:13:39.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 1.095044409s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
14932024-10-04T22:13:39.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 804.74305ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
14942024-10-04T22:13:39.147ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 806.967115ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
14952024-10-04T22:13:39.155ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.198004471s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
14962024-10-04T22:13:39.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.451033473s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
14972024-10-04T22:13:39.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 736.361081ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
14982024-10-04T22:13:39.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.115011663s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
14992024-10-04T22:13:39.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.338244702s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
15002024-10-04T22:13:39.372ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.150216293s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
15012024-10-04T22:13:39.606ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 697.618254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
15022024-10-04T22:13:39.614ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 764.901627ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
15032024-10-04T22:13:39.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.30271157s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
15042024-10-04T22:13:39.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 816.30323ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
15052024-10-04T22:13:39.776ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 768.268913ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
15062024-10-04T22:13:39.784ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 701.400196ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
15072024-10-04T22:13:39.798ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 990.592489ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
15082024-10-04T22:13:39.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 576.005539ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
15092024-10-04T22:13:39.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.101885909s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
15102024-10-04T22:13:39.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 523.018134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
15112024-10-04T22:13:40.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 737.567815ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
15122024-10-04T22:13:40.185ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 1.249168952s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
15132024-10-04T22:13:40.314ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.431400368s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
15142024-10-04T22:13:40.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.168206211s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
15152024-10-04T22:13:40.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 930.296718ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
15162024-10-04T22:13:40.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 704.134944ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
15172024-10-04T22:13:40.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 870.727519ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
15182024-10-04T22:13:40.544ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.490217137s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
15192024-10-04T22:13:40.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 1.464313949s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
15202024-10-04T22:13:40.561ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 982.865902ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
15212024-10-04T22:13:40.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 536.712776ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
15222024-10-04T22:13:40.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.473369525s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
15232024-10-04T22:13:40.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.149912364s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
15242024-10-04T22:13:40.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.016175864s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
15252024-10-04T22:13:40.777ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15262024-10-04T22:13:40.777ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15272024-10-04T22:13:40.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.498689877s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
15282024-10-04T22:13:40.839ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 867.204499ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
15292024-10-04T22:13:41.000ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 51.096142ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15302024-10-04T22:13:41.009ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 647.205041ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
15312024-10-04T22:13:41.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 67.852843ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15322024-10-04T22:13:41.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.22784032s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
15332024-10-04T22:13:41.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 1.116716404s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
15342024-10-04T22:13:41.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 641.07173ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
15352024-10-04T22:13:41.137ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 123.634984ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15362024-10-04T22:13:41.312ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 482.413418ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15372024-10-04T22:13:41.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 619.962736ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
15382024-10-04T22:13:41.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.313728153s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
15392024-10-04T22:13:41.486ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 716.171738ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
15402024-10-04T22:13:41.535ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.402211856s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
15412024-10-04T22:13:41.553ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.075162624s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
15422024-10-04T22:13:41.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.025266224s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
15432024-10-04T22:13:41.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.419053797s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
15442024-10-04T22:13:41.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.465093048s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
15452024-10-04T22:13:41.755ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 568.647901ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
15462024-10-04T22:13:41.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 879.489049ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
15472024-10-04T22:13:41.794ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.449011201s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
15482024-10-04T22:13:41.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 869.579115ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15492024-10-04T22:13:41.954ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 627.163549ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
15502024-10-04T22:13:42.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 954.798664ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
15512024-10-04T22:13:42.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 743.961881ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
15522024-10-04T22:13:42.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 555.571475ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
15532024-10-04T22:13:42.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 509.167655ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
15542024-10-04T22:13:42.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 676.501213ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
15552024-10-04T22:13:42.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 796.458396ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
15562024-10-04T22:13:42.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.41814355s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
15572024-10-04T22:13:42.333ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.323224472s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
15582024-10-04T22:13:42.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 731.093054ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
15592024-10-04T22:13:42.626ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.447378191s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
15602024-10-04T22:13:42.639ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 681.459906ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
15612024-10-04T22:13:42.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 659.396815ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
15622024-10-04T22:13:42.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 946.009009ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15632024-10-04T22:13:42.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 712.504813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
15642024-10-04T22:13:42.732ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 1.245317265s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
15652024-10-04T22:13:42.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 866.820826ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
15662024-10-04T22:13:42.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 569.296865ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
15672024-10-04T22:13:42.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 1.131831804s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
15682024-10-04T22:13:42.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.12307254s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
15692024-10-04T22:13:42.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 792.759438ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
15702024-10-04T22:13:43.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 537.926142ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
15712024-10-04T22:13:43.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 727.146276ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
15722024-10-04T22:13:43.192ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.024758762s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
15732024-10-04T22:13:43.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 569.200182ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
15742024-10-04T22:13:43.330ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.175231009s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
15752024-10-04T22:13:43.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 619.636982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
15762024-10-04T22:13:43.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.362756263s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
15772024-10-04T22:13:43.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 742.70183ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
15782024-10-04T22:13:43.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 697.651592ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
15792024-10-04T22:13:43.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 702.477055ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15802024-10-04T22:13:43.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.223524698s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
15812024-10-04T22:13:43.666ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 601.162678ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
15822024-10-04T22:13:43.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 720.626803ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
15832024-10-04T22:13:43.749ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 543.315103ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
15842024-10-04T22:13:43.799ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 623.416025ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
15852024-10-04T22:13:43.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.11933616s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
15862024-10-04T22:13:43.854ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 774.872618ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
15872024-10-04T22:13:43.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.222441011s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
15882024-10-04T22:13:43.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 1.191771833s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
15892024-10-04T22:13:44.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 1.382400739s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
15902024-10-04T22:13:44.085ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.309356758s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
15912024-10-04T22:13:44.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.345739845s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
15922024-10-04T22:13:44.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.226372578s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
15932024-10-04T22:13:44.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 767.514277ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
15942024-10-04T22:13:44.231ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 695.257097ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
15952024-10-04T22:13:44.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 632.342109ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
15962024-10-04T22:13:44.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 570.400784ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
15972024-10-04T22:13:44.361ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 715.429513ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
15982024-10-04T22:13:44.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.044429065s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
15992024-10-04T22:13:44.439ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 797.98968ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16002024-10-04T22:13:44.521ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.20189125s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
16012024-10-04T22:13:44.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 533.782199ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
16022024-10-04T22:13:44.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 514.501848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
16032024-10-04T22:13:44.890ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 991.80456ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
16042024-10-04T22:13:44.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.200688467s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
16052024-10-04T22:13:44.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 687.905725ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
16062024-10-04T22:13:44.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 518.263382ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
16072024-10-04T22:13:44.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.032337113s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
16082024-10-04T22:13:44.973ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.319480774s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
16092024-10-04T22:13:45.093ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 646.739695ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
16102024-10-04T22:13:45.169ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:3::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> <propval type="astring" name="ch_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="ch_binary" value='/opt/oxide//opt/oxide/clickhouse/clickhouse'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16112024-10-04T22:13:45.170ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16122024-10-04T22:13:45.196ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 635.877195ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
16132024-10-04T22:13:45.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 645.271484ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
16142024-10-04T22:13:45.229ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 550.619414ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
16152024-10-04T22:13:45.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 872.9152ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16162024-10-04T22:13:45.277ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.238756794s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
16172024-10-04T22:13:45.382ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 544.386702ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
16182024-10-04T22:13:45.410ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 975.989344ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
16192024-10-04T22:13:45.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 56.012571ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16202024-10-04T22:13:45.473ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.069980407s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
16212024-10-04T22:13:45.484ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 519.736331ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
16222024-10-04T22:13:45.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.375091745s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
16232024-10-04T22:13:45.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 587.999586ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
16242024-10-04T22:13:45.519ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 73.58607ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16252024-10-04T22:13:45.603ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 195.72773ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16262024-10-04T22:13:45.628ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 728.682674ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
16272024-10-04T22:13:45.892ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 749.951479ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
16282024-10-04T22:13:45.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 620.830586ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
16292024-10-04T22:13:45.915ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 271.363192ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16302024-10-04T22:13:45.923ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 935.408262ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
16312024-10-04T22:13:45.933ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.20706083s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
16322024-10-04T22:13:45.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 926.260942ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
16332024-10-04T22:13:45.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 686.568723ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
16342024-10-04T22:13:45.959ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 1.489550929s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
16352024-10-04T22:13:45.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 570.834447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
16362024-10-04T22:13:46.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 862.163592ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
16372024-10-04T22:13:46.103ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 606.977095ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
16382024-10-04T22:13:46.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 753.806545ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
16392024-10-04T22:13:46.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 526.542754ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16402024-10-04T22:13:46.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 472.786885ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16412024-10-04T22:13:46.304ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 936.085414ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
16422024-10-04T22:13:46.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 612.142431ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
16432024-10-04T22:13:46.397ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 619.383194ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
16442024-10-04T22:13:46.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.11266132s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
16452024-10-04T22:13:46.537ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.418005335s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
16462024-10-04T22:13:46.555ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.26514625s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
16472024-10-04T22:13:46.577ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.309291747s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
16482024-10-04T22:13:46.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 654.583584ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
16492024-10-04T22:13:46.669ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 753.828015ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
16502024-10-04T22:13:46.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 911.660344ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16512024-10-04T22:13:46.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 611.415856ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16522024-10-04T22:13:46.723ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 508.804234ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
16532024-10-04T22:13:46.873ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.291107842s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
16542024-10-04T22:13:46.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.453055996s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
16552024-10-04T22:13:46.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 815.702074ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
16562024-10-04T22:13:46.910ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.383479692s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
16572024-10-04T22:13:46.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 1.05123823s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
16582024-10-04T22:13:46.993ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 735.922026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
16592024-10-04T22:13:47.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 1.474745996s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
16602024-10-04T22:13:47.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 585.86043ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
16612024-10-04T22:13:47.249ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 716.512716ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
16622024-10-04T22:13:47.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.48874248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
16632024-10-04T22:13:47.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 1.314235027s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16642024-10-04T22:13:47.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 1.326527102s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
16652024-10-04T22:13:47.433ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.385705954s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
16662024-10-04T22:13:47.459ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 974.398356ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
16672024-10-04T22:13:47.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 545.692613ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16682024-10-04T22:13:47.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 777.143797ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
16692024-10-04T22:13:47.730ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.48594288s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
16702024-10-04T22:13:47.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.295944896s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
16712024-10-04T22:13:47.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 1.294096811s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
16722024-10-04T22:13:47.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.383360781s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
16732024-10-04T22:13:47.900ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.173231827s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
16742024-10-04T22:13:47.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.356178658s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
16752024-10-04T22:13:47.980ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 1.337742475s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
16762024-10-04T22:13:47.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 824.168938ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
16772024-10-04T22:13:48.168ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 703.614797ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16782024-10-04T22:13:48.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 846.62289ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
16792024-10-04T22:13:48.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271") failed: Property not found. retry in 1.333385762s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
16802024-10-04T22:13:48.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 920.178982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
16812024-10-04T22:13:48.444ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 991.519489ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
16822024-10-04T22:13:48.455ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 1.096554482s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
16832024-10-04T22:13:48.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 506.188894ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
16842024-10-04T22:13:48.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 1.221499569s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
16852024-10-04T22:13:48.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 993.098336ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
16862024-10-04T22:13:48.767ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a") failed: Property not found. retry in 1.134569967s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
16872024-10-04T22:13:48.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 1.193979146s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
16882024-10-04T22:13:48.843ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26") failed: Property not found. retry in 1.373535945s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
16892024-10-04T22:13:48.883ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 640.229534ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
16902024-10-04T22:13:49.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 830.821664ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
16912024-10-04T22:13:49.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 792.136798ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
16922024-10-04T22:13:49.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 591.711267ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
16932024-10-04T22:13:49.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 1.31961456s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
16942024-10-04T22:13:49.089ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 507.578249ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
16952024-10-04T22:13:49.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 500.017854ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
16962024-10-04T22:13:49.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 886.311381ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
16972024-10-04T22:13:49.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.299897782s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
16982024-10-04T22:13:49.327ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 994.098792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
16992024-10-04T22:13:49.336ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32") failed: Property not found. retry in 1.49216136s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
17002024-10-04T22:13:49.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 602.865036ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
17012024-10-04T22:13:49.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3") failed: Property not found. retry in 1.23757011s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
17022024-10-04T22:13:49.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 1.433740504s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
17032024-10-04T22:13:49.610ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85") failed: Property not found. retry in 1.354167544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
17042024-10-04T22:13:49.666ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52259
17052024-10-04T22:13:49.674ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7883 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52259 req_id = f49f8e4e-a9fa-40a6-b1a2-7f30f375a750 response_code = 200 uri = /baseboard
17062024-10-04T22:13:49.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7") failed: Property not found. retry in 884.935254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
17072024-10-04T22:13:49.696ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 875.550415ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
17082024-10-04T22:13:49.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b") failed: Property not found. retry in 1.070153446s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
17092024-10-04T22:13:49.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b") failed: Property not found. retry in 1.38331538s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
17102024-10-04T22:13:49.880ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf") failed: Property not found. retry in 879.434284ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
17112024-10-04T22:13:49.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 999.532341ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
17122024-10-04T22:13:50.044ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 648.731889ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
17132024-10-04T22:13:50.068ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856") failed: Property not found. retry in 1.319295626s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
17142024-10-04T22:13:50.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c") failed: Property not found. retry in 1.289902189s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
17152024-10-04T22:13:50.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d") failed: Property not found. retry in 980.330704ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
17162024-10-04T22:13:50.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3") failed: Property not found. retry in 689.345155ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
17172024-10-04T22:13:50.596ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206") failed: Property not found. retry in 809.420705ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
17182024-10-04T22:13:50.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520") failed: Property not found. retry in 1.414101631s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
17192024-10-04T22:13:50.703ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a") failed: Property not found. retry in 951.056142ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
17202024-10-04T22:13:50.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 1.232282318s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
17212024-10-04T22:13:51.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468") failed: Property not found. retry in 1.231379142s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
17222024-10-04T22:13:51.864ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 3.45496479s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 544b0af3-4a18-4467-8697-af690997796e
17232024-10-04T22:13:52.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 816.729634ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
17242024-10-04T22:13:53.001ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 1.492335044s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
17252024-10-04T22:13:54.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052") failed: Property not found. retry in 618.22975ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
17262024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_6436d814-77d3-42d3-9e54-70a16d185c32
17272024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_external_dns_e5972f58-0215-4789-9de6-8c745fc8691c
17282024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_5b35dee7-890d-4eb6-8ee1-7360a15c561a
17292024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_clickhouse_ba3eccf6-4897-41d8-a079-7fc999427052
17302024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_67428935-72b2-44d0-8f19-d288da37d55b
17312024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_230eb49b-a0c6-474e-aace-ba30ff793520
17322024-10-04T22:13:55.152ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_39899e1a-3f0a-429a-ae05-77f3247d3e26
17332024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_2367eac2-ed23-44dc-8c16-a31ddf5412c7
17342024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_f1bebc8f-e709-4c9a-a00c-cd503c04c45b
17352024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_46d21628-a828-4b09-b280-d1af35f56206
17362024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_external_dns_fcb39657-4027-45f8-9b5b-fc09e764f468
17372024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_nexus_e72dd7c9-9d72-4f49-b48b-231a411d26a3
17382024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_oximeter_75036a45-c39d-44e7-9208-1799441e9ebf
17392024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_pantry_98ea9840-3ff8-4efd-aa4d-cfad1addd77d
17402024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_66a90fa6-7dcc-4ced-93d6-67a7ab8c3856
17412024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_b6678fd5-9eca-4aa7-9e47-23ad27c73c85
17422024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_72974fd7-cc49-40bb-8097-19b6d7d747d3
17432024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_06c64b28-3935-456b-b9a9-8035576b6e7a
17442024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3288 zone = oxz_crucible_e74049ca-0d69-456c-8680-673308a2e271
17452024-10-04T22:13:55.153ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17462024-10-04T22:13:55.155ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
17472024-10-04T22:13:55.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 35517957 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:53624 req_id = 8cfc4482-f838-4688-a398-122caffee561 response_code = 204 uri = /omicron-zones
17482024-10-04T22:13:55.159ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1322
17492024-10-04T22:13:55.159ZINFOSledAgent (RSS): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
17502024-10-04T22:13:55.159ZINFOSledAgent (RSS): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
17512024-10-04T22:13:55.162ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:707
17522024-10-04T22:13:55.162ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:720
17532024-10-04T22:13:55.193ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:887
17542024-10-04T22:13:55.197ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17552024-10-04T22:13:55.353ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17562024-10-04T22:13:55.354ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 8.426512353s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers)" file = oximeter/producer/src/lib.rs:424 sled_id = 544b0af3-4a18-4467-8697-af690997796e
17572024-10-04T22:13:55.414ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17582024-10-04T22:13:55.531ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17592024-10-04T22:13:55.754ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17602024-10-04T22:13:56.002ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17612024-10-04T22:13:56.130ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17622024-10-04T22:13:56.301ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17632024-10-04T22:13:56.628ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17642024-10-04T22:13:57.024ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17652024-10-04T22:13:57.721ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17662024-10-04T22:13:58.038ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17672024-10-04T22:13:58.917ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17682024-10-04T22:13:59.806ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17692024-10-04T22:14:00.755ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17702024-10-04T22:14:02.043ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/4cb5ec19-6817-424a-9a2f-064289331fa0/initialization-complete) file = sled-agent/src/rack_setup/service.rs:949
17712024-10-04T22:14:03.576ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "46f55c15-8b63-4cb3-92d7-131cc97a8a1e", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:03 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"4cb5ec19-6817-424a-9a2f-064289331fa0\"", request_id: "46f55c15-8b63-4cb3-92d7-131cc97a8a1e" } file = sled-agent/src/rack_setup/service.rs:949
17722024-10-04T22:14:05.616ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "5c586af0-5676-412a-b5cc-ad60b3a6d2ef", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:05 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "5c586af0-5676-412a-b5cc-ad60b3a6d2ef" } file = sled-agent/src/rack_setup/service.rs:949
17732024-10-04T22:14:08.411ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "bbd8dda8-6e4a-48f0-93de-02dedad4a6fb", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:07 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "bbd8dda8-6e4a-48f0-93de-02dedad4a6fb" } file = sled-agent/src/rack_setup/service.rs:949
17742024-10-04T22:14:10.447ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "f615b1c7-566a-4a14-91c1-90acd59a8429", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:10 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "f615b1c7-566a-4a14-91c1-90acd59a8429" } file = sled-agent/src/rack_setup/service.rs:949
17752024-10-04T22:14:14.677ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "6ca851aa-6335-4c58-9655-14dc200f5bbd", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:13 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "6ca851aa-6335-4c58-9655-14dc200f5bbd" } file = sled-agent/src/rack_setup/service.rs:949
17762024-10-04T22:14:19.525ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "d6a8f74a-e0bb-45b2-abbb-e31f1ffad2da", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:19 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "d6a8f74a-e0bb-45b2-abbb-e31f1ffad2da" } file = sled-agent/src/rack_setup/service.rs:949
17772024-10-04T22:14:19.709ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:54528
17782024-10-04T22:14:19.710ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 29 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:54528 req_id = 35993060-edfc-4520-aec3-895fdb70c256 response_code = 200 uri = /baseboard
17792024-10-04T22:14:23.067ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "65c80e27-b678-4ef0-bb7a-e944c1c137f0", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:23 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "65c80e27-b678-4ef0-bb7a-e944c1c137f0" } file = sled-agent/src/rack_setup/service.rs:949
17802024-10-04T22:14:28.611ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "e0069a4d-2815-4c4c-98da-6f9c6de96788", "content-length": "173", "date": "Fri, 04 Oct 2024 22:14:28 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: sled with id \"544b0af3-4a18-4467-8697-af690997796e\"", request_id: "e0069a4d-2815-4c4c-98da-6f9c6de96788" } file = sled-agent/src/rack_setup/service.rs:949
17812024-10-04T22:14:33.980ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:50448 remote_addr = [fd00:1122:3344:101::d]:62949 sled_id = 544b0af3-4a18-4467-8697-af690997796e
17822024-10-04T22:14:34.100ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119201 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 54cb7ba3-fd47-4d42-a85a-d07fd7e4e74c response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
17832024-10-04T22:14:35.795ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = 544b0af3-4a18-4467-8697-af690997796e
17842024-10-04T22:14:35.904ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38581
17852024-10-04T22:14:35.905ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17862024-10-04T22:14:36.130ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 226571 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38581 req_id = fbd4c237-0b92-442b-b966-c1b3565a089f response_code = 200 uri = /inventory
17872024-10-04T22:14:36.135ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17882024-10-04T22:14:36.135ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
17892024-10-04T22:14:36.135ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
17902024-10-04T22:14:36.136ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 873 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38581 req_id = fc86ea99-38be-4b97-a2be-bbb8d2513cb0 response_code = 200 uri = /omicron-zones
17912024-10-04T22:14:38.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61235
17922024-10-04T22:14:38.809ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46b99b09-11d4-4fe4-bf41-6ed61ba36d44, kind: Service { id: e5972f58-0215-4789-9de6-8c745fc8691c }, name: Name("external-dns-e5972f58-0215-4789-9de6-8c745fc8691c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 244])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bbfb69f6-617d-4782-a1f5-f7e225f6c330, kind: Service { id: fcb39657-4027-45f8-9b5b-fc09e764f468 }, name: Name("external-dns-fcb39657-4027-45f8-9b5b-fc09e764f468"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 178])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e67383-e8ea-42d3-b048-d94c625ebeb6, kind: Service { id: e72dd7c9-9d72-4f49-b48b-231a411d26a3 }, name: Name("nexus-e72dd7c9-9d72-4f49-b48b-231a411d26a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 146])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9195d467-8fcf-4fcc-9e20-edc728f0400f, kind: Service { id: 46d21628-a828-4b09-b280-d1af35f56206 }, name: Name("nexus-46d21628-a828-4b09-b280-d1af35f56206"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948c7b04-5bd4-4111-a486-7fc541ba08f2, kind: Service { id: 5b35dee7-890d-4eb6-8ee1-7360a15c561a }, name: Name("nexus-5b35dee7-890d-4eb6-8ee1-7360a15c561a"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 148, 10])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
17932024-10-04T22:14:38.809ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17942024-10-04T22:14:38.810ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17952024-10-04T22:14:38.810ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17962024-10-04T22:14:38.810ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
17972024-10-04T22:14:38.810ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
17982024-10-04T22:14:38.810ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
17992024-10-04T22:14:38.810ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 14085 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61235 req_id = 11513ef3-d706-4f06-8c93-5782bafa2d0d response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18002024-10-04T22:14:38.811ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:959
18012024-10-04T22:14:38.867ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45277
18022024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46b99b09-11d4-4fe4-bf41-6ed61ba36d44, kind: Service { id: e5972f58-0215-4789-9de6-8c745fc8691c }, name: Name("external-dns-e5972f58-0215-4789-9de6-8c745fc8691c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 244])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bbfb69f6-617d-4782-a1f5-f7e225f6c330, kind: Service { id: fcb39657-4027-45f8-9b5b-fc09e764f468 }, name: Name("external-dns-fcb39657-4027-45f8-9b5b-fc09e764f468"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 178])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e67383-e8ea-42d3-b048-d94c625ebeb6, kind: Service { id: e72dd7c9-9d72-4f49-b48b-231a411d26a3 }, name: Name("nexus-e72dd7c9-9d72-4f49-b48b-231a411d26a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 146])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9195d467-8fcf-4fcc-9e20-edc728f0400f, kind: Service { id: 46d21628-a828-4b09-b280-d1af35f56206 }, name: Name("nexus-46d21628-a828-4b09-b280-d1af35f56206"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948c7b04-5bd4-4111-a486-7fc541ba08f2, kind: Service { id: 5b35dee7-890d-4eb6-8ee1-7360a15c561a }, name: Name("nexus-5b35dee7-890d-4eb6-8ee1-7360a15c561a"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 148, 10])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18032024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18042024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18052024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18062024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18072024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18082024-10-04T22:14:38.868ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
18092024-10-04T22:14:38.868ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1038 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45277 req_id = ec36c3d3-d59d-4d1e-ac2c-a489153a383f response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18102024-10-04T22:14:38.920ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55716
18112024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46b99b09-11d4-4fe4-bf41-6ed61ba36d44, kind: Service { id: e5972f58-0215-4789-9de6-8c745fc8691c }, name: Name("external-dns-e5972f58-0215-4789-9de6-8c745fc8691c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 244])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bbfb69f6-617d-4782-a1f5-f7e225f6c330, kind: Service { id: fcb39657-4027-45f8-9b5b-fc09e764f468 }, name: Name("external-dns-fcb39657-4027-45f8-9b5b-fc09e764f468"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 178])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e67383-e8ea-42d3-b048-d94c625ebeb6, kind: Service { id: e72dd7c9-9d72-4f49-b48b-231a411d26a3 }, name: Name("nexus-e72dd7c9-9d72-4f49-b48b-231a411d26a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 146])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9195d467-8fcf-4fcc-9e20-edc728f0400f, kind: Service { id: 46d21628-a828-4b09-b280-d1af35f56206 }, name: Name("nexus-46d21628-a828-4b09-b280-d1af35f56206"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948c7b04-5bd4-4111-a486-7fc541ba08f2, kind: Service { id: 5b35dee7-890d-4eb6-8ee1-7360a15c561a }, name: Name("nexus-5b35dee7-890d-4eb6-8ee1-7360a15c561a"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 148, 10])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18122024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18132024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18142024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18152024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18162024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18172024-10-04T22:14:38.920ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
18182024-10-04T22:14:38.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 228 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:55716 req_id = 271e2b76-d706-4e15-bfb1-d8617fb01aa4 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18192024-10-04T22:14:39.142ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48030
18202024-10-04T22:14:39.142ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18212024-10-04T22:14:39.354ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 211926 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48030 req_id = c0aad4a8-f700-43d5-871a-92bb820a3924 response_code = 200 uri = /inventory
18222024-10-04T22:14:39.355ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18232024-10-04T22:14:39.356ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/config/omicron-zones.json file = common/src/ledger.rs:177
18242024-10-04T22:14:39.356ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/config/omicron-zones.json file = common/src/ledger.rs:177
18252024-10-04T22:14:39.356ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1402 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48030 req_id = 51989889-f546-4976-83de-a318aa8ff2e6 response_code = 200 uri = /omicron-zones
18262024-10-04T22:14:40.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53248
18272024-10-04T22:14:40.779ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 46b99b09-11d4-4fe4-bf41-6ed61ba36d44, kind: Service { id: e5972f58-0215-4789-9de6-8c745fc8691c }, name: Name("external-dns-e5972f58-0215-4789-9de6-8c745fc8691c"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 158, 244])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: bbfb69f6-617d-4782-a1f5-f7e225f6c330, kind: Service { id: fcb39657-4027-45f8-9b5b-fc09e764f468 }, name: Name("external-dns-fcb39657-4027-45f8-9b5b-fc09e764f468"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 137, 178])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 13e67383-e8ea-42d3-b048-d94c625ebeb6, kind: Service { id: e72dd7c9-9d72-4f49-b48b-231a411d26a3 }, name: Name("nexus-e72dd7c9-9d72-4f49-b48b-231a411d26a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 174, 146])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 9195d467-8fcf-4fcc-9e20-edc728f0400f, kind: Service { id: 46d21628-a828-4b09-b280-d1af35f56206 }, name: Name("nexus-46d21628-a828-4b09-b280-d1af35f56206"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 203, 186])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 948c7b04-5bd4-4111-a486-7fc541ba08f2, kind: Service { id: 5b35dee7-890d-4eb6-8ee1-7360a15c561a }, name: Name("nexus-5b35dee7-890d-4eb6-8ee1-7360a15c561a"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 148, 10])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
18282024-10-04T22:14:40.780ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18292024-10-04T22:14:40.780ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18302024-10-04T22:14:40.780ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18312024-10-04T22:14:40.780ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
18322024-10-04T22:14:40.780ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
18332024-10-04T22:14:40.780ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
18342024-10-04T22:14:40.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 248 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53248 req_id = c2438283-1004-4153-9390-db95f727667c response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18352024-10-04T22:14:48.587ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41453
18362024-10-04T22:14:48.593ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4946 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41453 req_id = ea23e109-f825-4ce1-b738-62e58b7402da response_code = 200 uri = /vpc-routes
18372024-10-04T22:14:48.666ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 812 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:41453 req_id = 98cbc411-a99c-4feb-ac47-f66bdbb61efc response_code = 204 uri = /vpc-routes
18382024-10-04T22:14:49.744ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:62346
18392024-10-04T22:14:49.744ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 37 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:62346 req_id = 6eba4686-ff21-4cb5-9d8b-73ca8a103fbd response_code = 200 uri = /baseboard
18402024-10-04T22:15:03.981ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8281 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 65a3dd73-bac0-449f-92c3-b482d3c69584 response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
18412024-10-04T22:15:04.563ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59353
18422024-10-04T22:15:04.563ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59353 req_id = 3f25950a-774e-4d5e-a9b1-6088e8d46bd5 response_code = 200 uri = /vpc-routes
18432024-10-04T22:15:04.595ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50338
18442024-10-04T22:15:04.596ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1372 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50338 req_id = 876f250f-bec6-4e82-9520-ac4e8bb5d723 response_code = 200 uri = /v2p
18452024-10-04T22:15:04.623ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:59353 req_id = 638f60a0-fc93-4251-aace-ac2cff5697c7 response_code = 204 uri = /vpc-routes
18462024-10-04T22:15:04.725ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43787
18472024-10-04T22:15:04.725ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18482024-10-04T22:15:04.807ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18492024-10-04T22:15:04.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 151326 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43787 req_id = 719476af-e136-47e3-a83f-fb2579153966 response_code = 204 uri = /switch-ports
18502024-10-04T22:15:04.876ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41645
18512024-10-04T22:15:04.877ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41645 req_id = 18878e76-c344-46a6-9e10-ce1b48a6ba0a response_code = 200 uri = /vpc-routes
18522024-10-04T22:15:04.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 2739 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43787 req_id = 3b24bf3b-62f2-4275-ae0f-23d4afbd4fc2 response_code = 200 uri = /network-bootstore-config
18532024-10-04T22:15:04.894ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41645 req_id = 293709e8-b2e0-4161-a16b-f962efd4e808 response_code = 204 uri = /vpc-routes
18542024-10-04T22:15:04.955ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
18552024-10-04T22:15:04.955ZINFOSledAgent (bootstore): Writing ledger to /pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18562024-10-04T22:15:04.956ZINFOSledAgent (bootstore): Writing ledger to /pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
18572024-10-04T22:15:04.957ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
18582024-10-04T22:15:04.957ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3271 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:43787 req_id = cbbf5bd8-467f-4035-9c59-0f37428c0d1e response_code = 204 uri = /network-bootstore-config
18592024-10-04T22:15:04.961ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38314
18602024-10-04T22:15:04.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 72 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38314 req_id = 802a7dca-b1ef-4521-b3c6-c7728bfada13 response_code = 200 uri = /vpc-routes
18612024-10-04T22:15:04.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:38314 req_id = 155292b5-2ddc-4e91-8e6b-15a453665f5c response_code = 204 uri = /vpc-routes
18622024-10-04T22:15:05.096ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58873
18632024-10-04T22:15:05.097ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58873 req_id = 201d83d4-fc7e-49c2-9eca-3dbe81c16a5b response_code = 200 uri = /v2p
18642024-10-04T22:15:05.122ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35601
18652024-10-04T22:15:05.122ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35601 req_id = 84603bcb-3aaa-4621-8661-24a5ea2d3042 response_code = 200 uri = /v2p
18662024-10-04T22:15:05.223ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40773
18672024-10-04T22:15:05.223ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18682024-10-04T22:15:05.261ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18692024-10-04T22:15:05.317ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94008 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:40773 req_id = 79939896-6ca4-42c8-9c12-02fa58cbed26 response_code = 204 uri = /switch-ports
18702024-10-04T22:15:05.317ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46844
18712024-10-04T22:15:05.317ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18722024-10-04T22:15:05.355ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18732024-10-04T22:15:05.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98647 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46844 req_id = 94d8cfbb-bac3-4d10-8414-657b0e28d947 response_code = 204 uri = /switch-ports
18742024-10-04T22:15:05.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40773 req_id = 613dfd86-4201-4138-848f-37fa56ac2cb4 response_code = 200 uri = /network-bootstore-config
18752024-10-04T22:15:05.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46844 req_id = 72a9ec95-2faa-4a7d-823b-a6c4b3c91547 response_code = 200 uri = /network-bootstore-config
18762024-10-04T22:15:19.777ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:65522
18772024-10-04T22:15:19.777ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 67 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:65522 req_id = 61f4c057-9df8-4e60-a3d0-bc4e4f2ae64d response_code = 200 uri = /baseboard
18782024-10-04T22:15:33.982ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9414 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = c71c34a8-666b-46c3-b7a3-b9802cade6a9 response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
18792024-10-04T22:15:34.015ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61284
18802024-10-04T22:15:34.015ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 56 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61284 req_id = a369f9a4-7974-47b9-9d53-d3c813325131 response_code = 200 uri = /vpc-routes
18812024-10-04T22:15:34.030ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 135 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61284 req_id = 38faba5e-204c-4065-a39a-975efc59f342 response_code = 204 uri = /vpc-routes
18822024-10-04T22:15:34.399ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:62028
18832024-10-04T22:15:34.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 163 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:62028 req_id = 5afadb22-c045-4f33-b3e5-542c185908e9 response_code = 200 uri = /v2p
18842024-10-04T22:15:34.401ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34616
18852024-10-04T22:15:34.402ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18862024-10-04T22:15:34.440ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18872024-10-04T22:15:34.496ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94729 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:34616 req_id = ed4be7e7-ae00-4472-9d26-81373134cf8e response_code = 204 uri = /switch-ports
18882024-10-04T22:15:34.497ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 119 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34616 req_id = bee715ef-3688-4eba-bf58-c3ac6ea27abb response_code = 200 uri = /network-bootstore-config
18892024-10-04T22:15:34.762ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34269
18902024-10-04T22:15:34.763ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 108 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34269 req_id = d2748fc6-a9b9-42dd-8d20-846c6a1c9ee2 response_code = 200 uri = /vpc-routes
18912024-10-04T22:15:34.774ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34269 req_id = 0d382460-ba5f-48cf-9b23-1a00d288f63f response_code = 204 uri = /vpc-routes
18922024-10-04T22:15:34.780ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33274
18932024-10-04T22:15:34.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33274 req_id = 5279aea6-aa74-48dc-b91c-67f699b903d2 response_code = 200 uri = /vpc-routes
18942024-10-04T22:15:34.789ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 91 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33274 req_id = cc72d0c6-8470-4931-bc1a-be1e43f4e50a response_code = 204 uri = /vpc-routes
18952024-10-04T22:15:34.869ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46614
18962024-10-04T22:15:34.869ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
18972024-10-04T22:15:34.906ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
18982024-10-04T22:15:34.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96543 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46614 req_id = 251706be-e395-4e13-970b-8f66238ee574 response_code = 204 uri = /switch-ports
18992024-10-04T22:15:34.966ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53970
19002024-10-04T22:15:34.966ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62742
19012024-10-04T22:15:34.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 307 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46614 req_id = 7240834e-2901-441a-8280-5673868087de response_code = 200 uri = /network-bootstore-config
19022024-10-04T22:15:34.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 310 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53970 req_id = def4656d-d4ef-422c-ae03-937b70ec9f15 response_code = 200 uri = /v2p
19032024-10-04T22:15:34.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 469 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62742 req_id = 6944094b-d47e-44e4-b4de-b06d7fe528f6 response_code = 200 uri = /v2p
19042024-10-04T22:15:34.996ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48422
19052024-10-04T22:15:34.996ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19062024-10-04T22:15:35.032ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19072024-10-04T22:15:35.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 94137 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:48422 req_id = 4902b2ad-8a7f-4d11-b17c-c3f2b5613a90 response_code = 204 uri = /switch-ports
19082024-10-04T22:15:35.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 202 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48422 req_id = c6692039-411f-4154-a4fb-43636626b6e5 response_code = 200 uri = /network-bootstore-config
19092024-10-04T22:15:49.812ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34468
19102024-10-04T22:15:49.812ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 31 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34468 req_id = 8473f3c9-b3fc-4bd5-91b3-bfa8343749e5 response_code = 200 uri = /baseboard
19112024-10-04T22:16:03.980ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8025 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 082dfc8e-f627-46b9-908f-94b58e9a7b0d response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
19122024-10-04T22:16:04.243ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35204
19132024-10-04T22:16:04.243ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35204 req_id = 30c53757-0b14-44ad-98cc-c18cf4b17b44 response_code = 200 uri = /vpc-routes
19142024-10-04T22:16:04.252ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:35204 req_id = 0c1afa58-286f-4c39-83ce-a7b258646ebb response_code = 204 uri = /vpc-routes
19152024-10-04T22:16:04.399ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44813
19162024-10-04T22:16:04.399ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44813 req_id = b9cc1a30-9036-4b38-a9bb-92c05b33d66e response_code = 200 uri = /v2p
19172024-10-04T22:16:04.413ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50749
19182024-10-04T22:16:04.414ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19192024-10-04T22:16:04.455ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19202024-10-04T22:16:04.511ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97787 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50749 req_id = d6f6926e-f072-4c7e-a6d3-be0aaa26b2df response_code = 204 uri = /switch-ports
19212024-10-04T22:16:04.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50749 req_id = a7a3985f-6be2-4c67-8649-13202c16d13f response_code = 200 uri = /network-bootstore-config
19222024-10-04T22:16:04.849ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48406
19232024-10-04T22:16:04.850ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48406 req_id = 97def262-301a-4039-973b-9ded852ae0af response_code = 200 uri = /vpc-routes
19242024-10-04T22:16:04.866ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 179 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:48406 req_id = 314700d6-71d5-4bb5-82ff-224646eb226b response_code = 204 uri = /vpc-routes
19252024-10-04T22:16:04.905ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59405
19262024-10-04T22:16:04.905ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59405 req_id = fca304e3-c149-4de1-bd7c-4457647c0e2e response_code = 200 uri = /vpc-routes
19272024-10-04T22:16:04.934ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59405 req_id = 75794ef3-7161-4bb2-a5ed-a630c8ceb78c response_code = 204 uri = /vpc-routes
19282024-10-04T22:16:05.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44939
19292024-10-04T22:16:05.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44939 req_id = e69e5f2d-cf4b-42fa-ac2b-463ff57cfbfa response_code = 200 uri = /v2p
19302024-10-04T22:16:05.145ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:32793
19312024-10-04T22:16:05.146ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19322024-10-04T22:16:05.184ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19332024-10-04T22:16:05.244ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98319 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:32793 req_id = dcb2ffd1-3d1d-44d2-b9ba-ff10ff4926a9 response_code = 204 uri = /switch-ports
19342024-10-04T22:16:05.244ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33061
19352024-10-04T22:16:05.244ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46967
19362024-10-04T22:16:05.244ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 320 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:32793 req_id = e28aed46-8e1e-411c-ac0c-99e9a55de614 response_code = 200 uri = /network-bootstore-config
19372024-10-04T22:16:05.244ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19382024-10-04T22:16:05.244ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 321 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33061 req_id = 84d68fed-910b-43a7-aea5-0e72ae49085c response_code = 200 uri = /v2p
19392024-10-04T22:16:05.285ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19402024-10-04T22:16:05.343ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98485 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46967 req_id = d191d698-79ca-4298-ac4a-a742ae1ad7bf response_code = 204 uri = /switch-ports
19412024-10-04T22:16:05.343ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46967 req_id = 02f8131a-7c4a-436e-b608-29b0c62420de response_code = 200 uri = /network-bootstore-config
19422024-10-04T22:16:19.844ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52621
19432024-10-04T22:16:19.844ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 38 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52621 req_id = 800888e0-6b9c-40a2-ac83-8668b5714b07 response_code = 200 uri = /baseboard
19442024-10-04T22:16:33.984ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6346 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 719d0399-6257-496f-b95e-55455804bb9a response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
19452024-10-04T22:16:34.064ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39732
19462024-10-04T22:16:34.064ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39732 req_id = b456874e-cd49-41b3-9719-96effaf61159 response_code = 200 uri = /vpc-routes
19472024-10-04T22:16:34.111ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38030
19482024-10-04T22:16:34.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 116 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38030 req_id = 9c619008-4eab-4e49-9149-189f6d3f6ba4 response_code = 200 uri = /v2p
19492024-10-04T22:16:34.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 69 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:39732 req_id = 31f5968e-cac1-41f1-b473-f31e95deaa52 response_code = 204 uri = /vpc-routes
19502024-10-04T22:16:34.165ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43093
19512024-10-04T22:16:34.165ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19522024-10-04T22:16:34.201ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19532024-10-04T22:16:34.259ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93634 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43093 req_id = 1aff5caa-3844-4d61-8fba-6e7aff3ea2db response_code = 204 uri = /switch-ports
19542024-10-04T22:16:34.259ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43093 req_id = 89a388f1-f7cf-4f8f-9ba6-f0e3f1c2807b response_code = 200 uri = /network-bootstore-config
19552024-10-04T22:16:34.771ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55386
19562024-10-04T22:16:34.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 90 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:55386 req_id = 51956cf7-291c-4f85-b180-f743ef473302 response_code = 200 uri = /vpc-routes
19572024-10-04T22:16:34.786ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:45173
19582024-10-04T22:16:34.786ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:45173 req_id = af0d6319-15c9-4e08-8e3a-810751003083 response_code = 200 uri = /vpc-routes
19592024-10-04T22:16:34.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 114 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:45173 req_id = 81bcad50-d378-4a74-8d86-d94827f7ea56 response_code = 204 uri = /vpc-routes
19602024-10-04T22:16:34.843ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46083
19612024-10-04T22:16:34.843ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46083 req_id = f7b47982-2466-4127-931a-347a1fa8fd07 response_code = 200 uri = /v2p
19622024-10-04T22:16:34.847ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60493
19632024-10-04T22:16:34.847ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60493 req_id = da92993c-8f57-4ed9-a7ac-fc4f96dde69a response_code = 200 uri = /v2p
19642024-10-04T22:16:34.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 76 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55386 req_id = 3767c203-0b2f-44c7-8453-c487dfde72fc response_code = 204 uri = /vpc-routes
19652024-10-04T22:16:34.886ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47272
19662024-10-04T22:16:34.886ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19672024-10-04T22:16:34.922ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19682024-10-04T22:16:34.978ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92567 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47272 req_id = 138a0b99-bd39-4a4e-accb-d91b49e17efe response_code = 204 uri = /switch-ports
19692024-10-04T22:16:34.979ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44432
19702024-10-04T22:16:34.979ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 268 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47272 req_id = 8d02f5f5-9a35-4839-87f0-9f71f8bbbdba response_code = 200 uri = /network-bootstore-config
19712024-10-04T22:16:34.980ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19722024-10-04T22:16:35.016ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19732024-10-04T22:16:35.077ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 98258 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44432 req_id = de330024-4422-4eac-b94e-17820569b86e response_code = 204 uri = /switch-ports
19742024-10-04T22:16:35.078ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 171 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44432 req_id = e3606ce7-4e14-4b07-9c3f-a108d5618ae7 response_code = 200 uri = /network-bootstore-config
19752024-10-04T22:16:49.877ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64855
19762024-10-04T22:16:49.877ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64855 req_id = 1e1d9d2a-d135-4bd5-b999-394ffd479f6e response_code = 200 uri = /baseboard
19772024-10-04T22:17:04.064ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6689 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 710ce5ab-bc2c-4900-a3a2-0039da601c79 response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
19782024-10-04T22:17:04.117ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48863
19792024-10-04T22:17:04.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 54 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48863 req_id = d557047c-cdd5-487a-8b78-c95d761fce03 response_code = 200 uri = /vpc-routes
19802024-10-04T22:17:04.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48863 req_id = 4aafe8bd-c3b4-4df1-939e-86dcf974ed68 response_code = 204 uri = /vpc-routes
19812024-10-04T22:17:04.341ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37527
19822024-10-04T22:17:04.341ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37527 req_id = 5ab399eb-21d3-439e-939c-482f3a554669 response_code = 200 uri = /v2p
19832024-10-04T22:17:04.533ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:40290
19842024-10-04T22:17:04.533ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
19852024-10-04T22:17:04.589ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
19862024-10-04T22:17:04.700ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 167849 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:40290 req_id = 270ec054-a674-4797-b9e5-0423c0bd01fd response_code = 204 uri = /switch-ports
19872024-10-04T22:17:04.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:40290 req_id = a1d871ee-512f-4353-a16d-41adec1bc3fe response_code = 200 uri = /network-bootstore-config
19882024-10-04T22:17:04.906ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45962
19892024-10-04T22:17:04.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45962 req_id = dd4f1ad6-1ddf-41eb-9436-dd76cd846598 response_code = 200 uri = /vpc-routes
19902024-10-04T22:17:04.922ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:45962 req_id = 0bb83321-da7c-411e-9d03-fd377fef2594 response_code = 204 uri = /vpc-routes
19912024-10-04T22:17:04.976ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33644
19922024-10-04T22:17:04.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33644 req_id = 3f8415bc-3d2f-4ce5-85a1-e2703a277f62 response_code = 200 uri = /vpc-routes
19932024-10-04T22:17:04.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1016 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33644 req_id = baadfb39-6808-4f8f-a2dc-59189b56bb99 response_code = 204 uri = /vpc-routes
19942024-10-04T22:17:05.101ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43708
19952024-10-04T22:17:05.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 127 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43708 req_id = 6857ddd0-2298-44d9-86a6-5527c87b5537 response_code = 200 uri = /v2p
19962024-10-04T22:17:05.161ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38253
19972024-10-04T22:17:05.162ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 899 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38253 req_id = 66a646ae-1107-440e-bfbc-f778f384e107 response_code = 200 uri = /v2p
19982024-10-04T22:17:05.199ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57190
19992024-10-04T22:17:05.199ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20002024-10-04T22:17:05.236ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20012024-10-04T22:17:05.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97796 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57190 req_id = 242a9a5a-7fdf-4cfc-98ca-992f6c86fd92 response_code = 204 uri = /switch-ports
20022024-10-04T22:17:05.297ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42592
20032024-10-04T22:17:05.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 189 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57190 req_id = a8d1e9b0-eb73-498f-adca-a055caa508d3 response_code = 200 uri = /network-bootstore-config
20042024-10-04T22:17:05.299ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20052024-10-04T22:17:05.341ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20062024-10-04T22:17:05.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 103360 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:42592 req_id = 7d1e5109-68fd-4d21-8813-641b813d7142 response_code = 204 uri = /switch-ports
20072024-10-04T22:17:05.402ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42592 req_id = b67c8bd4-ddbb-4e47-aab4-be9c7e3d17bb response_code = 200 uri = /network-bootstore-config
20082024-10-04T22:17:07.108ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
20092024-10-04T22:17:19.909ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61702
20102024-10-04T22:17:19.910ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 770 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61702 req_id = fa3c8edb-9cfe-44db-a84f-7663fe1f1bfc response_code = 200 uri = /baseboard
20112024-10-04T22:17:34.065ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 7551 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 19cc1caf-1058-4773-9311-b3a6296cf245 response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
20122024-10-04T22:17:34.116ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:58789
20132024-10-04T22:17:34.116ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 52 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:58789 req_id = 1a1776c1-39b3-4404-a2fa-f78c76f2be8b response_code = 200 uri = /vpc-routes
20142024-10-04T22:17:34.124ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 752 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:58789 req_id = c48c4039-e885-4cb5-85fd-551ae48c79fd response_code = 204 uri = /vpc-routes
20152024-10-04T22:17:34.297ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36533
20162024-10-04T22:17:34.297ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 134 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36533 req_id = 1dbe2199-5400-45a9-8e6a-5f6e7008ce84 response_code = 200 uri = /v2p
20172024-10-04T22:17:34.333ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65359
20182024-10-04T22:17:34.333ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20192024-10-04T22:17:34.372ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20202024-10-04T22:17:34.426ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 93255 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:65359 req_id = 7a6a29bb-c857-41a9-ab15-275b9aea8b0b response_code = 204 uri = /switch-ports
20212024-10-04T22:17:34.427ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 666 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65359 req_id = 069f33e1-6870-4ca6-8ef2-e98c1ab72844 response_code = 200 uri = /network-bootstore-config
20222024-10-04T22:17:34.729ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39261
20232024-10-04T22:17:34.729ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 694 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39261 req_id = eb48cf26-cd83-4be1-8931-1438746293ec response_code = 200 uri = /vpc-routes
20242024-10-04T22:17:34.745ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 183 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:39261 req_id = 94a54650-3cc4-4154-bc64-ce02314942e7 response_code = 204 uri = /vpc-routes
20252024-10-04T22:17:34.851ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56027
20262024-10-04T22:17:34.851ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56027 req_id = 6b1dfb24-390c-4295-8b43-06fec9801e8c response_code = 200 uri = /vpc-routes
20272024-10-04T22:17:34.861ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56027 req_id = dd1441ba-bc68-40ee-89c7-c86ae2c2999d response_code = 204 uri = /vpc-routes
20282024-10-04T22:17:34.905ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56841
20292024-10-04T22:17:34.905ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20302024-10-04T22:17:34.947ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20312024-10-04T22:17:35.016ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110998 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:56841 req_id = c018a58b-0e59-469b-ad0f-1a736362a4ba response_code = 204 uri = /switch-ports
20322024-10-04T22:17:35.016ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57645
20332024-10-04T22:17:35.016ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44370
20342024-10-04T22:17:35.016ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 282 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56841 req_id = 1c63541b-57ec-4ab2-8aa2-964718778b8f response_code = 200 uri = /network-bootstore-config
20352024-10-04T22:17:35.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 914 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57645 req_id = 1880e841-db71-40f9-8153-b586d846f71e response_code = 200 uri = /v2p
20362024-10-04T22:17:35.017ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 945 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44370 req_id = cbe9bc48-1815-4a62-b9ab-7cb1e0e5c4f8 response_code = 200 uri = /v2p
20372024-10-04T22:17:35.094ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53964
20382024-10-04T22:17:35.094ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20392024-10-04T22:17:35.135ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20402024-10-04T22:17:35.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 99470 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53964 req_id = 86577c90-e388-4273-be9f-e152b061ae4b response_code = 204 uri = /switch-ports
20412024-10-04T22:17:35.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53964 req_id = 580e8e35-2f49-4978-98ad-9ec2c1b3bbae response_code = 200 uri = /network-bootstore-config
20422024-10-04T22:17:49.943ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:52277
20432024-10-04T22:17:49.943ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 34 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:52277 req_id = 890d327d-bda2-468b-bed5-192047ef0c2f response_code = 200 uri = /baseboard
20442024-10-04T22:18:04.066ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 8846 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = d5edb636-86b9-4f27-ba76-b32a0233d538 response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
20452024-10-04T22:18:04.185ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60821
20462024-10-04T22:18:04.185ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60821 req_id = 0c0ed0cf-a75d-4bfb-be16-2f8a9bd8c809 response_code = 200 uri = /vpc-routes
20472024-10-04T22:18:04.201ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60821 req_id = b90482a9-3096-49d3-8ddd-96c6e24ea053 response_code = 204 uri = /vpc-routes
20482024-10-04T22:18:04.382ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36657
20492024-10-04T22:18:04.382ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36657 req_id = 81328ef6-290b-4ff7-9f08-b4381bd1fcde response_code = 200 uri = /v2p
20502024-10-04T22:18:04.466ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56228
20512024-10-04T22:18:04.466ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20522024-10-04T22:18:04.506ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20532024-10-04T22:18:04.572ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 106358 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56228 req_id = 5d03c33b-e905-4b26-b83a-47a761b83441 response_code = 204 uri = /switch-ports
20542024-10-04T22:18:04.573ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56228 req_id = c853237c-182e-465e-8c32-88c9ff616a7d response_code = 200 uri = /network-bootstore-config
20552024-10-04T22:18:04.853ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55981
20562024-10-04T22:18:04.853ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55981 req_id = 5927b573-ac78-413d-8181-99d6173afa6e response_code = 200 uri = /vpc-routes
20572024-10-04T22:18:04.862ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:55981 req_id = d5c545eb-ab15-4d34-8e3d-daf86a1137bb response_code = 204 uri = /vpc-routes
20582024-10-04T22:18:04.876ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41167
20592024-10-04T22:18:04.876ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 38 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41167 req_id = 5ab00613-0dae-4ac1-ac56-7f61f74458c7 response_code = 200 uri = /vpc-routes
20602024-10-04T22:18:04.897ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41167 req_id = 50c3da74-ecda-49ea-b46c-b33e98d0c10f response_code = 204 uri = /vpc-routes
20612024-10-04T22:18:04.986ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58096
20622024-10-04T22:18:04.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58096 req_id = 393f8b7f-5777-4fcc-95a1-f28cd1397a2b response_code = 200 uri = /v2p
20632024-10-04T22:18:05.077ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44723
20642024-10-04T22:18:05.078ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 100 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44723 req_id = 3a4a9895-7cb4-4b63-ae38-d184ab0e0942 response_code = 200 uri = /v2p
20652024-10-04T22:18:05.108ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56020
20662024-10-04T22:18:05.109ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20672024-10-04T22:18:05.147ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20682024-10-04T22:18:05.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 97919 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56020 req_id = 45f46709-7a84-4081-ab6c-dc881bb127ef response_code = 204 uri = /switch-ports
20692024-10-04T22:18:05.206ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51912
20702024-10-04T22:18:05.207ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
20712024-10-04T22:18:05.210ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3820 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56020 req_id = 9b3b7bd3-98ea-4765-b2bc-944e736b2b85 response_code = 200 uri = /network-bootstore-config
20722024-10-04T22:18:05.241ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
20732024-10-04T22:18:05.296ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 89538 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:51912 req_id = 8579b858-872c-4a1b-843f-165136492b45 response_code = 204 uri = /switch-ports
20742024-10-04T22:18:05.296ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 177 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51912 req_id = 83db7a9b-7bd7-4a53-affc-0feb25100aa4 response_code = 200 uri = /network-bootstore-config
20752024-10-04T22:18:19.981ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37893
20762024-10-04T22:18:19.981ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 125 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37893 req_id = dee40319-f196-4e98-9c04-35d10cfa61a5 response_code = 200 uri = /baseboard
20772024-10-04T22:18:30.670ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48104
20782024-10-04T22:18:30.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48104 req_id = b41c856f-782e-48fc-8877-3f2abe163dc9 response_code = 200 uri = /v2p
20792024-10-04T22:18:30.671ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11189461) }
20802024-10-04T22:18:30.672ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1176 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:48104 req_id = 37645259-48bd-4c72-8a5c-a4aaf1c1861c response_code = 204 uri = /v2p
20812024-10-04T22:18:30.694ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43170
20822024-10-04T22:18:30.700ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:539 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(11189461))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-47b66cf83a0", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 13b1c897-dd1d-468d-8cd2-a87d61528446, read_only_parent: Some(Volume { block_size: 512, id: 200b226b-f62f-47b5-b339-f7667a1f01b2, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 5960cd9d-eb4b-4806-bea7-ffbe09847b66, key: Some("G/RG1hoFO0rmvV13S0E58Bj3KRbDEgHhvHQ8HIxSddw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 13b1c897-dd1d-468d-8cd2-a87d61528446, key: Some("Ddm1/2ubus7NgbTwAxnQXEkaLc7VBduOhK+Bk8TZE78="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::1a]:19003", "[fd00:1122:3344:101::16]:19000"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-47b66cf83a0" }] }), cloud_init_bytes: Some(..) } instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 metadata = InstanceMetadata { silo_id: 5782a804-afa1-460e-b170-bf8495d7f170, project_id: a2a6f842-3219-4148-9113-865ec5e05329 } migration_id = None propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-04T22:18:30.692471595Z }
20832024-10-04T22:18:30.700ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:572 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 migration_id = None propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
20842024-10-04T22:18:30.700ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:1113 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 migration_id = None propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(11189461))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 5dbca880-7d32-4498-8f99-b01c004d248f, kind: Instance { id: 0d966f86-2001-459d-9023-a8291a8f49d9 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 240, 0, 0])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11189461), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-47b66cf83a0", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 13b1c897-dd1d-468d-8cd2-a87d61528446, read_only_parent: Some(Volume { block_size: 512, id: 200b226b-f62f-47b5-b339-f7667a1f01b2, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 5960cd9d-eb4b-4806-bea7-ffbe09847b66, key: Some("G/RG1hoFO0rmvV13S0E58Bj3KRbDEgHhvHQ8HIxSddw="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::1a]:19002", "[fd00:1122:3344:101::12]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 13b1c897-dd1d-468d-8cd2-a87d61528446, key: Some("Ddm1/2ubus7NgbTwAxnQXEkaLc7VBduOhK+Bk8TZE78="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::19]:19002", "[fd00:1122:3344:101::1a]:19003", "[fd00:1122:3344:101::16]:19000"] } }] } }], boot_settings: Some(BootSettings { order: [BootOrderEntry { name: "disk-47b66cf83a0" }] }), cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-10-04T22:18:30.692471595Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
20852024-10-04T22:18:30.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 6371 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:43170 req_id = 0b0e30ba-f0b7-46a1-9124-c918d29a0409 response_code = 200 uri = /vmms/2cc38d1a-5eae-4208-baf7-b332dc17e588
20862024-10-04T22:18:30.831ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42929
20872024-10-04T22:18:30.833ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1230 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42929 req_id = 46620f53-2b8b-4e6d-ba59-ec8940e92108 response_code = 200 uri = /vmms/2cc38d1a-5eae-4208-baf7-b332dc17e588/state
20882024-10-04T22:18:30.833ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 240, 0, 0]), slot: 0, vni: Vni { inner: 11189461 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
20892024-10-04T22:18:30.834ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
20902024-10-04T22:18:30.871ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588 file = illumos-utils/src/zone.rs:316 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
20912024-10-04T22:18:30.892ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588 file = illumos-utils/src/zone.rs:354 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
20922024-10-04T22:18:32.220ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
20932024-10-04T22:18:32.220ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588 zone file = illumos-utils/src/running_zone.rs:488 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
20942024-10-04T22:18:32.416ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 47.355558ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
20952024-10-04T22:18:32.474ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 92.812345ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
20962024-10-04T22:18:32.582ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 286.237573ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
20972024-10-04T22:18:32.883ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 222.465948ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
20982024-10-04T22:18:33.119ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 445.980388ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
20992024-10-04T22:18:33.576ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 563.560541ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
21002024-10-04T22:18:34.045ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57430
21012024-10-04T22:18:34.066ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 9104 local_addr = [fd00:1122:3344:101::1]:50448 method = GET remote_addr = [fd00:1122:3344:101::d]:62949 req_id = 3446c617-931b-4b67-bef7-9289a3aafafa response_code = 200 sled_id = 544b0af3-4a18-4467-8697-af690997796e uri = /544b0af3-4a18-4467-8697-af690997796e
21022024-10-04T22:18:34.152ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 987.610828ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
21032024-10-04T22:18:34.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44282
21042024-10-04T22:18:34.176ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 940 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44282 req_id = a4abfee3-49b4-4798-b5f7-11a906c13e6a response_code = 200 uri = /vpc-routes
21052024-10-04T22:18:34.247ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63175
21062024-10-04T22:18:34.247ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63175 req_id = 62cce18a-7dd0-4faa-b64f-e12cc31fda0a response_code = 200 uri = /v2p
21072024-10-04T22:18:34.293ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35802
21082024-10-04T22:18:34.293ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
21092024-10-04T22:18:34.332ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
21102024-10-04T22:18:34.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 101296 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35802 req_id = 505304fd-a612-4c9e-9049-ee70cc96e88a response_code = 204 uri = /switch-ports
21112024-10-04T22:18:34.394ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35802 req_id = 52441021-4488-4f9c-adce-d9fe55df31bc response_code = 200 uri = /network-bootstore-config
21122024-10-04T22:18:34.410ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 326 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44282 req_id = 300d28c5-30ef-4edc-8175-6dae48379252 response_code = 204 uri = /vpc-routes
21132024-10-04T22:18:34.716ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46148
21142024-10-04T22:18:34.781ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64685
21152024-10-04T22:18:34.781ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64685 req_id = ca0f46d2-2650-40ec-828e-49637ac5deb9 response_code = 200 uri = /vpc-routes
21162024-10-04T22:18:34.802ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:64685 req_id = 109db59a-8014-4a9e-bbb5-93731a23a18c response_code = 204 uri = /vpc-routes
21172024-10-04T22:18:34.821ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:38625
21182024-10-04T22:18:34.872ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47095
21192024-10-04T22:18:34.873ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
21202024-10-04T22:18:34.911ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
21212024-10-04T22:18:34.965ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 92708 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47095 req_id = d2bad2a9-618e-4030-9ab1-da72c5ef1c11 response_code = 204 uri = /switch-ports
21222024-10-04T22:18:34.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36087
21232024-10-04T22:18:34.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44388
21242024-10-04T22:18:34.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38854
21252024-10-04T22:18:34.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 310 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47095 req_id = 11f0f4e7-8182-43b3-9d21-9641a442721d response_code = 200 uri = /network-bootstore-config
21262024-10-04T22:18:34.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 241 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38854 req_id = b12c57c0-66c9-4c13-b4c3-f2867a572faa response_code = 200 uri = /v2p
21272024-10-04T22:18:34.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1003 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44388 req_id = 957aa318-6444-479f-9e5a-7e42a29fdee7 response_code = 200 uri = /vpc-routes
21282024-10-04T22:18:34.967ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 1164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36087 req_id = b3ae9446-36ea-4960-a1b5-1abc3ea8afac response_code = 200 uri = /v2p
21292024-10-04T22:18:34.983ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:44388 req_id = a6e6b543-7dc6-4c84-b6fe-01ed0379010f response_code = 204 uri = /vpc-routes
21302024-10-04T22:18:34.991ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58917
21312024-10-04T22:18:34.991ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4096
21322024-10-04T22:18:35.030ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4120
21332024-10-04T22:18:35.087ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 96056 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58917 req_id = f34e2a31-e940-4461-af8b-cd3e19804a7a response_code = 204 uri = /switch-ports
21342024-10-04T22:18:35.087ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58917 req_id = 8f409932-62ff-4c69-850c-276a678b5223 response_code = 200 uri = /network-bootstore-config
21352024-10-04T22:18:35.150ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 904.198313ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
21362024-10-04T22:18:36.067ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 844.855799ms file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
21372024-10-04T22:18:36.923ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588") failed: Property not found. retry in 1.312233479s file = illumos-utils/src/svc.rs:36 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 zone = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
21382024-10-04T22:18:38.259ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588 file = sled-agent/src/instance.rs:1594 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21392024-10-04T22:18:38.266ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1603 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21402024-10-04T22:18:38.302ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1627 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21412024-10-04T22:18:38.437ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:825 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21422024-10-04T22:18:38.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 4392753 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57430 req_id = 8394c4a9-4b93-4ece-ba9c-6a8b66e54011 response_code = 200 uri = /vmms/2cc38d1a-5eae-4208-baf7-b332dc17e588/state
21432024-10-04T22:18:38.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3722190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46148 req_id = 6c333f75-f39e-4810-8e1e-52b48b1c44b7 response_code = 200 uri = /vmms/2cc38d1a-5eae-4208-baf7-b332dc17e588/state
21442024-10-04T22:18:38.438ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 3617010 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:38625 req_id = 8abd510f-e7f6-44ff-abfd-4f4a3ad8c99b response_code = 200 uri = /vmms/2cc38d1a-5eae-4208-baf7-b332dc17e588/state
21452024-10-04T22:18:38.443ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-10-04T22:18:38.443433381Z } file = sled-agent/src/instance.rs:675 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21462024-10-04T22:18:38.443ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:697 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-04T22:18:38.443433381Z } propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21472024-10-04T22:18:38.443ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:595 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(3), time_updated: 2024-10-04T22:18:38.443433381Z }, migration_in: None, migration_out: None }
21482024-10-04T22:18:38.480ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-10-04T22:18:38.480912180Z } file = sled-agent/src/instance.rs:675 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21492024-10-04T22:18:38.480ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:697 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-04T22:18:38.480912180Z } propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21502024-10-04T22:18:38.480ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:595 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(4), time_updated: 2024-10-04T22:18:38.480912180Z }, migration_in: None, migration_out: None }
21512024-10-04T22:18:48.359ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54744
21522024-10-04T22:18:48.359ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 478 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:54744 req_id = e7d92cc1-cb04-43fc-8272-0ed3b5b57ed7 response_code = 200 uri = /vmms/2cc38d1a-5eae-4208-baf7-b332dc17e588/state
21532024-10-04T22:18:48.359ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-10-04T22:18:48.359931923Z } file = sled-agent/src/instance.rs:675 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21542024-10-04T22:18:48.359ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:697 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-04T22:18:48.359931923Z } propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21552024-10-04T22:18:48.360ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:595 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(6), time_updated: 2024-10-04T22:18:48.359931923Z }, migration_in: None, migration_out: None }
21562024-10-04T22:18:48.372ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-10-04T22:18:48.372350241Z } file = sled-agent/src/instance.rs:675 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21572024-10-04T22:18:48.372ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:697 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-10-04T22:18:48.372350241Z } propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21582024-10-04T22:18:48.372ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:711 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21592024-10-04T22:18:48.372ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21602024-10-04T22:18:48.372ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 15472056240820932054 n_samples = 6 sled_id = 544b0af3-4a18-4467-8697-af690997796e
21612024-10-04T22:18:48.372ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:602 id = 9249817892003710266 n_samples = 6 sled_id = 544b0af3-4a18-4467-8697-af690997796e
21622024-10-04T22:18:48.375ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
21632024-10-04T22:18:48.375ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/86b9b4ff-6295-44c9-a6e7-e01d8a2e11b1/debug/bundle/zone", "/pool/int/f89edb19-85d1-4199-b7bf-a91aa5ecc8e8/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/676a3716-3d3e-465b-9699-cf98062dabca/crypt/debug", "/pool/ext/9af95e51-a064-462a-bea1-8d3042c749de/crypt/debug", "/pool/ext/9e3d11a1-af30-407b-b05f-d5f1a5762d41/crypt/debug", "/pool/ext/ac4d5d54-7085-42e4-a0d6-359bebb3c91f/crypt/debug", "/pool/ext/c4a31058-0731-4581-8d9d-c5c4b66770da/crypt/debug", "/pool/ext/75662bd1-8970-44ab-88a5-02e715476179/crypt/debug", "/pool/ext/1279c2fc-c95f-45e2-a67c-783305b4023a/crypt/debug", "/pool/ext/d4135165-e105-4373-9d12-4efe0b6e4c50/crypt/debug", "/pool/ext/2ca4e6da-c9a6-43af-84fc-f0b4a5fc0987/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588
21642024-10-04T22:18:48.998ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1142 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588", bundle_id: 1b3f9fe3-1739-41f7-bff1-ad8c2e24d85c }, time_created: 2024-10-04T22:18:48.375265069Z, version: 0, cause: TerminatedInstance }
21652024-10-04T22:18:48.998ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_2cc38d1a-5eae-4208-baf7-b332dc17e588 file = sled-agent/src/instance.rs:925 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21662024-10-04T22:18:50.019ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53360
21672024-10-04T22:18:50.019ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 41 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53360 req_id = de49bbb6-914d-4515-8102-81b7966dad52 response_code = 200 uri = /baseboard
21682024-10-04T22:18:50.739ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21692024-10-04T22:18:50.755ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(11189461), kind: System }
21702024-10-04T22:18:50.755ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(11189461), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
21712024-10-04T22:18:50.756ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:595 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-04T22:18:50.756048085Z }, migration_in: None, migration_out: None }
21722024-10-04T22:18:50.806ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:595 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588 state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(8), time_updated: 2024-10-04T22:18:50.756048085Z }, migration_in: None, migration_out: None }
21732024-10-04T22:18:50.810ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:862 instance_id = 0d966f86-2001-459d-9023-a8291a8f49d9 propolis_id = 2cc38d1a-5eae-4208-baf7-b332dc17e588
21742024-10-04T22:18:51.193ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43271
21752024-10-04T22:18:51.194ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 563 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43271 req_id = 979fab9c-4122-4b6e-8ff3-fb49667046a1 response_code = 200 uri = /vpc-routes
21762024-10-04T22:18:51.202ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:43271 req_id = 6f491288-300b-46be-84ac-05aa4936d6f2 response_code = 204 uri = /vpc-routes
21772024-10-04T22:18:51.368ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35026
21782024-10-04T22:18:51.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 122 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35026 req_id = 0d17b8b9-4b3f-4838-9c27-822e5c54a509 response_code = 200 uri = /v2p
21792024-10-04T22:18:51.487ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:48572
21802024-10-04T22:18:51.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:48572 req_id = 3fa07cbd-31d5-4fb3-8256-67e460464ba7 response_code = 200 uri = /v2p
21812024-10-04T22:18:52.127ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:1105 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42259
21822024-10-04T22:18:52.127ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 645 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:42259 req_id = 5aba5b9d-ca02-4e49-85cb-b7ba2efe2700 response_code = 200 uri = /vpc-routes
21832024-10-04T22:18:52.135ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dropshot-0.12.0/src/server.rs:950 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42259 req_id = 1816e643-2bb7-4af5-93ba-2e3a359db154 response_code = 204 uri = /vpc-routes