Max level shown:
1 [ Aug 29 15:14:13 Enabled. ]
2 [ Aug 29 15:14:13 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Aug 29 15:14:13 Method "start" exited with status 0. ]
4 [ Aug 29 15:14:13 Rereading configuration. ]
5 [ Aug 29 15:14:13 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-08-29T15:14:13.281ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:113
82024-08-29T15:14:13.281ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-08-29T15:14:13.333ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-08-29T15:14:13.333ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-08-29T15:14:13.446ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-08-29T15:14:13.455ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-08-29T15:14:13.463ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-08-29T15:14:14.066ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fdb0:18c0:4d0d:a02a::, width: 64 }
152024-08-29T15:14:14.203ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:126
162024-08-29T15:14:14.203ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:138
172024-08-29T15:14:14.203ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:151
182024-08-29T15:14:14.203ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:173 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-08-29T15:14:14.203ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:520
202024-08-29T15:14:14.203ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:716
212024-08-29T15:14:14.206ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:623
222024-08-29T15:14:14.206ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:765
232024-08-29T15:14:14.206ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:187
242024-08-29T15:14:14.206ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:824
252024-08-29T15:14:14.206ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_0.vdev
262024-08-29T15:14:14.206ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = None
272024-08-29T15:14:14.206ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
282024-08-29T15:14:14.206ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-08-29T15:14:14.206ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-08-29T15:14:14.218ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:360
312024-08-29T15:14:14.218ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool)
322024-08-29T15:14:14.318ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal }
332024-08-29T15:14:14.732ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
342024-08-29T15:14:14.732ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/m2_1.vdev
352024-08-29T15:14:14.732ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-08-29T15:14:14.732ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-08-29T15:14:14.732ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
382024-08-29T15:14:14.732ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-08-29T15:14:14.732ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-08-29T15:14:14.741ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:360
412024-08-29T15:14:14.741ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:371 id = 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool)
422024-08-29T15:14:14.829ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal }
432024-08-29T15:14:15.233ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_0.vdev
442024-08-29T15:14:15.233ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
452024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
462024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
472024-08-29T15:14:15.233ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
482024-08-29T15:14:15.233ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-08-29T15:14:15.233ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_1.vdev
502024-08-29T15:14:15.233ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
512024-08-29T15:14:15.233ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
522024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
532024-08-29T15:14:15.233ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
542024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
552024-08-29T15:14:15.233ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_2.vdev
562024-08-29T15:14:15.233ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
572024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
582024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
592024-08-29T15:14:15.233ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
602024-08-29T15:14:15.233ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
612024-08-29T15:14:15.233ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_3.vdev
622024-08-29T15:14:15.233ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
632024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
642024-08-29T15:14:15.233ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
652024-08-29T15:14:15.233ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
662024-08-29T15:14:15.233ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-08-29T15:14:15.233ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_4.vdev
682024-08-29T15:14:15.233ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
692024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
702024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
712024-08-29T15:14:15.234ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
722024-08-29T15:14:15.234ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
732024-08-29T15:14:15.234ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_5.vdev
742024-08-29T15:14:15.234ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
752024-08-29T15:14:15.234ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
762024-08-29T15:14:15.234ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
772024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
782024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
792024-08-29T15:14:15.234ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_6.vdev
802024-08-29T15:14:15.234ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
812024-08-29T15:14:15.234ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
822024-08-29T15:14:15.234ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
832024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
842024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
852024-08-29T15:14:15.234ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_7.vdev
862024-08-29T15:14:15.234ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
872024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
882024-08-29T15:14:15.234ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
892024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
902024-08-29T15:14:15.234ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-08-29T15:14:15.234ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:243 vdev = /scratch/u2_8.vdev
922024-08-29T15:14:15.234ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }), tx: ... } file = sled-storage/src/manager.rs:451
932024-08-29T15:14:15.234ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:457 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
942024-08-29T15:14:15.234ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
952024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
962024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
972024-08-29T15:14:15.234ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:97
982024-08-29T15:14:15.234ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:99
992024-08-29T15:14:15.234ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
1002024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
1012024-08-29T15:14:15.234ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
1022024-08-29T15:14:15.234ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1032024-08-29T15:14:15.234ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:211
1042024-08-29T15:14:15.234ZINFOSledAgent (bootstore): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-08-29T15:14:15.234ZINFOSledAgent (bootstore): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-08-29T15:14:15.234ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-08-29T15:14:15.234ZINFOSledAgent (bootstore): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-08-29T15:14:15.234ZINFOSledAgent (bootstore): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-08-29T15:14:15.234ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-08-29T15:14:15.234ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:216
1112024-08-29T15:14:15.234ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:231
1122024-08-29T15:14:15.245ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:730
1132024-08-29T15:14:15.245ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 20773, tv_nsec: 845925025 } time_to_next_cleanup = 599.999922868s
1142024-08-29T15:14:15.245ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3866
1152024-08-29T15:14:15.245ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1162024-08-29T15:14:15.245ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-08-29T15:14:15.245ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-08-29T15:14:15.248ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-08-29T15:14:15.248ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-08-29T15:14:15.248ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1053
1212024-08-29T15:14:15.248ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1222024-08-29T15:14:15.248ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1232024-08-29T15:14:15.248ZINFOSledAgent (RSS): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-08-29T15:14:15.248ZINFOSledAgent (RSS): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-08-29T15:14:15.248ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1113
1262024-08-29T15:14:15.248ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1272024-08-29T15:14:15.248ZINFOSledAgent (RSS): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-08-29T15:14:15.248ZINFOSledAgent (RSS): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-08-29T15:14:15.248ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1154
1302024-08-29T15:14:15.248ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:155
1312024-08-29T15:14:15.248ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:128
1322024-08-29T15:14:15.248ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1332024-08-29T15:14:15.248ZINFOSledAgent (RSS): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-08-29T15:14:15.249ZINFOSledAgent (RSS): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-08-29T15:14:15.249ZINFOSledAgent (RSS): Sled plan written to storage: Plan { rack_id: 0c368d89-20a1-4c09-9eea-581bb60b3770, sleds: { [fdb0:18c0:4d0d:a02a::1]:12346: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: b9c40332-a231-42db-adf0-b267d663a311, rack_id: 0c368d89-20a1-4c09-9eea-581bb60b3770, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64, }, }, }, }, }, config: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, local_pref: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, }, } file = sled-agent/src/rack_setup/plan/sled.rs:175
1362024-08-29T15:14:15.249ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1187
1372024-08-29T15:14:15.249ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-08-29T15:14:15.249ZINFOSledAgent (bootstore): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-08-29T15:14:15.249ZINFOSledAgent (bootstore): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-08-29T15:14:15.249ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-08-29T15:14:15.249ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: b9c40332-a231-42db-adf0-b267d663a311, rack_id: 0c368d89-20a1-4c09-9eea-581bb60b3770, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-08-29T15:14:15.249ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:56082
1432024-08-29T15:14:15.249ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: b9c40332-a231-42db-adf0-b267d663a311, rack_id: 0c368d89-20a1-4c09-9eea-581bb60b3770, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:359
1442024-08-29T15:14:15.249ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:374
1452024-08-29T15:14:15.249ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:451
1472024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:644
1482024-08-29T15:14:15.249ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1492024-08-29T15:14:15.249ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:101::, width: 64 }
1502024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1512024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_disks_ledger
1522024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:633
1532024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:661
1542024-08-29T15:14:15.249ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:440
1552024-08-29T15:14:15.250ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
1562024-08-29T15:14:15.250ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:532 result = DisksManagementResult { status: [] }
1572024-08-29T15:14:15.251ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:4163
1582024-08-29T15:14:15.251ZINFOSledAgent (ServiceManager): Starting switch zone file = sled-agent/src/services.rs:4545
1592024-08-29T15:14:15.282ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1602024-08-29T15:14:15.304ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:316
1612024-08-29T15:14:15.322ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1622024-08-29T15:14:15.329ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:388 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1632024-08-29T15:14:15.329ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
1642024-08-29T15:14:15.329ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:400 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1652024-08-29T15:14:15.329ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1662024-08-29T15:14:15.332ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 140.138782ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dcbaef70-cc44-449f-9aa0-1536ebd05f68", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dcbaef70-cc44-449f-9aa0-1536ebd05f68" } file = clients/ddm-admin-client/src/lib.rs:177
1672024-08-29T15:14:15.342ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1682024-08-29T15:14:15.365ZINFOSledAgent: successfully created encrypted zvol: name="oxi_5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1692024-08-29T15:14:15.365ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/swap" file = sled-agent/src/swap_device.rs:96
1702024-08-29T15:14:15.370ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:415 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1712024-08-29T15:14:15.370ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:127
1722024-08-29T15:14:15.370ZINFOSledAgent (BackingFs): Ensuring dataset oxi_5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/backing/fmd file = sled-agent/src/backing_fs.rs:137
1732024-08-29T15:14:15.457ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:172
1742024-08-29T15:14:15.472ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1752024-08-29T15:14:15.473ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 167.056344ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "85b79dfd-1f9f-4fd6-a68d-b81e28e79b3d", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "85b79dfd-1f9f-4fd6-a68d-b81e28e79b3d" } file = clients/ddm-admin-client/src/lib.rs:177
1762024-08-29T15:14:15.641ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1772024-08-29T15:14:15.641ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 215.738517ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d904bc70-8fb3-4292-a2c5-9d0e2439db74", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d904bc70-8fb3-4292-a2c5-9d0e2439db74" } file = clients/ddm-admin-client/src/lib.rs:177
1782024-08-29T15:14:15.677ZINFOSledAgent (BackingFs): Mounting oxi_5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:180
1792024-08-29T15:14:15.690ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:188
1802024-08-29T15:14:15.690ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:188
1812024-08-29T15:14:15.690ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:188
1822024-08-29T15:14:15.690ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:194
1832024-08-29T15:14:15.858ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1842024-08-29T15:14:15.858ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 173.70694ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "27a347bd-c311-4bb2-99fa-9ebac4737b0a", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "27a347bd-c311-4bb2-99fa-9ebac4737b0a" } file = clients/ddm-admin-client/src/lib.rs:177
1852024-08-29T15:14:16.033ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1862024-08-29T15:14:16.034ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 192.531334ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ef50c20e-a68e-4d4c-ab2e-79fe73fa9b19", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ef50c20e-a68e-4d4c-ab2e-79fe73fa9b19" } file = clients/ddm-admin-client/src/lib.rs:177
1872024-08-29T15:14:16.228ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1882024-08-29T15:14:16.228ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 193.130365ms) DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "11076fbe-32f5-4ffe-b028-ef655211089e", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "11076fbe-32f5-4ffe-b028-ef655211089e" } file = clients/ddm-admin-client/src/lib.rs:177
1892024-08-29T15:14:16.417ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1902024-08-29T15:14:16.422ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:34899 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1912024-08-29T15:14:16.422ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1922024-08-29T15:14:16.422ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:34899 file = oximeter/producer/src/lib.rs:283 interval = 30s producer_id = b9c40332-a231-42db-adf0-b267d663a311 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1932024-08-29T15:14:16.422ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:163
1942024-08-29T15:14:16.423ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 221.993246ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1952024-08-29T15:14:16.424ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1962024-08-29T15:14:16.424ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = b9c40332-a231-42db-adf0-b267d663a311
1972024-08-29T15:14:16.424ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:914 underlay_address = fd00:1122:3344:101::1
1982024-08-29T15:14:16.424ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:73
1992024-08-29T15:14:16.648ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 635.202892ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2002024-08-29T15:14:17.285ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 1.388018331s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2012024-08-29T15:14:18.675ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.207316312s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2022024-08-29T15:14:18.908ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2639
2032024-08-29T15:14:18.908ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2857
2042024-08-29T15:14:18.908ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2509
2052024-08-29T15:14:18.908ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2556
2062024-08-29T15:14:18.908ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2938
2072024-08-29T15:14:18.908ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2980
2082024-08-29T15:14:18.910ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2782
2092024-08-29T15:14:18.912ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2550
2102024-08-29T15:14:18.912ZINFOSledAgent (ServiceManager): Profile for oxz_switch: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/mgs"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12225'/> <propval type="astring" name="id" value='8f88c1d9-b275-491d-b7f9-340e22f40c0e'/> <propval type="astring" name="rack_id" value='0c368d89-20a1-4c09-9eea-581bb60b3770'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/wicketd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="address" value='[::1]:12226'/> <propval type="astring" name="artifact-address" value='[fdb0:18c0:4d0d:a02a::2]:12227'/> <propval type="astring" name="baseboard-file" value='/opt/oxide/baseboard.json'/> <propval type="astring" name="mgs-address" value='[::1]:12225'/> <propval type="astring" name="nexus-proxy-address" value='[::]:12229'/> <propval type="astring" name="rack-subnet" value='fd00:1122:3344::'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/switch_zone_setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="gz_local_link_addr" value='fe80::8:20ff:fe0e:242c'/> <property type="astring" name="link_local_links"> <astring_list> <value_node value="tfportrear0_0" /> <value_node value="oxBootstrap0" /> </astring_list> </property> <propval type="astring" name="bootstrap_addr" value='fdb0:18c0:4d0d:a02a::2'/> <propval type="astring" name="bootstrap_vnic" value='oxBootstrap0'/> <propval type="astring" name="baseboard_info" value='{"type":"pc","identifier":"sock","model":"i86pc"}'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/dendrite"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_id" value='b9c40332-a231-42db-adf0-b267d663a311'/> <propval type="astring" name="rack_id" value='0c368d89-20a1-4c09-9eea-581bb60b3770'/> <propval type="astring" name="address" value='[::1]:12224'/> <propval type="astring" name="front_ports" value='1'/> <propval type="astring" name="rear_ports" value='1'/> <propval type="astring" name="port_config" value='/opt/oxide/dendrite/misc/softnpu_single_sled_config.toml'/> <propval type="astring" name="mgmt" value='uds'/> <propval type="astring" name="uds_path" value='/opt/softnpu/stuff'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/tfport"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="host" value='[::1]'/> <propval type="astring" name="port" value='12224'/> <propval type="astring" name="flags" value='--sync-only'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/lldpd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="board_rev" value='softnpu_front_1_rear_1'/> <propval type="astring" name="scrimlet_id" value='sock'/> <propval type="astring" name="scrimlet_model" value='i86pc'/> <propval type="astring" name="address" value='[::1]:12230'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/pumpkind"> </service> <service version="1" type="service" name="oxide/mgd"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="sled_uuid" value='b9c40332-a231-42db-adf0-b267d663a311'/> <propval type="astring" name="rack_uuid" value='0c368d89-20a1-4c09-9eea-581bb60b3770'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/mg-ddm"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="mode" value='transit'/> <propval type="astring" name="dendrite" value='true'/> <propval type="astring" name="sled_uuid" value='b9c40332-a231-42db-adf0-b267d663a311'/> <propval type="astring" name="rack_uuid" value='0c368d89-20a1-4c09-9eea-581bb60b3770'/> <propval type="astring" name="interfaces" value='tfportrear0_0/ll'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/uplink"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
2112024-08-29T15:14:18.912ZINFOSledAgent (ServiceManager): Booting oxz_switch zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_switch
2122024-08-29T15:14:19.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 67.843637ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2132024-08-29T15:14:19.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 94.944248ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2142024-08-29T15:14:19.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 242.809621ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2152024-08-29T15:14:19.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 380.232042ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2162024-08-29T15:14:20.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 640.299106ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2172024-08-29T15:14:20.884ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 3.880013918s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2182024-08-29T15:14:21.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 683.73984ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2192024-08-29T15:14:22.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 1.115131775s file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2202024-08-29T15:14:23.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 718.587864ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2212024-08-29T15:14:24.697ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:205 local_addr = [fd00:1122:3344:101::1]:12345
2222024-08-29T15:14:24.697ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:429
2232024-08-29T15:14:24.698ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2242024-08-29T15:14:24.698ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2252024-08-29T15:14:24.698ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2262024-08-29T15:14:24.698ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2272024-08-29T15:14:24.698ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:271 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None, local_pref: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, lldp: None }], bgp: [], bfd: [] })))
2282024-08-29T15:14:24.698ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3866
2292024-08-29T15:14:24.698ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:56082
2302024-08-29T15:14:24.953ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2312024-08-29T15:14:24.953ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2322024-08-29T15:14:24.953ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2332024-08-29T15:14:24.953ZINFOSledAgent (RSS): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2342024-08-29T15:14:24.953ZINFOSledAgent (RSS): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/rss-service-plan-v4.json file = common/src/ledger.rs:188
2352024-08-29T15:14:24.953ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2362024-08-29T15:14:24.953ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2372024-08-29T15:14:24.953ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2382024-08-29T15:14:24.954ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.957320105s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:391 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2392024-08-29T15:14:24.958ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:4187 new = [fd00:1122:3344:101::2, ::1] old = [::1]
2402024-08-29T15:14:24.958ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:4203
2412024-08-29T15:14:24.958ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:534 zone = oxz_switch
2422024-08-29T15:14:25.221ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49015
2432024-08-29T15:14:25.222ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2442024-08-29T15:14:25.222ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 321 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49015 req_id = 634692d2-72df-49b4-b5c2-ca8fbbfea75c response_code = 200 uri = /inventory
2452024-08-29T15:14:25.253ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:36314
2462024-08-29T15:14:25.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:36314 req_id = 30678255-9da8-417b-af45-c5b384621f5d response_code = 200 uri = /sled-role
2472024-08-29T15:14:25.254ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
2482024-08-29T15:14:25.254ZINFOSledAgent (RSS): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2492024-08-29T15:14:25.254ZINFOSledAgent (RSS): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.rss-service-plan-v4.json.tmp file = common/src/ledger.rs:199
2502024-08-29T15:14:25.255ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:999
2512024-08-29T15:14:25.284ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 38bc4c99-f824-42b1-9252-61e07ec12c50, pool_id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: d2db14f3-ed37-4067-aabb-56412ba4cb84, pool_id: 866d818f-82bb-444c-992a-1351dec91efa (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 4ebd9cb5-27f4-4c65-94c6-ff793dd0343a, pool_id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 4c637c8b-6729-4597-a52c-8e37a3ad27e4, pool_id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: ce05b51a-cd89-4ebb-b688-efb500dd174e, pool_id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: c01d4db6-8364-4378-8e25-1d180755f647, pool_id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 4b4a6b84-2a5f-42e2-9e64-42ad4b741866, pool_id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: ab80f8c3-3ca5-4ec8-99ed-275ba79ffbf5, pool_id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 87286297-b93c-4248-91f6-2951af5888f5, pool_id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool) }] } file = sled-agent/src/rack_setup/service.rs:355 sled_address = [fd00:1122:3344:101::1]:12345
2522024-08-29T15:14:25.284ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63462
2532024-08-29T15:14:25.285ZINFOSledAgent: physical disks ensure file = sled-agent/src/sled_agent.rs:856 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2542024-08-29T15:14:25.285ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 38bc4c99-f824-42b1-9252-61e07ec12c50, pool_id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: d2db14f3-ed37-4067-aabb-56412ba4cb84, pool_id: 866d818f-82bb-444c-992a-1351dec91efa (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 4ebd9cb5-27f4-4c65-94c6-ff793dd0343a, pool_id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 4c637c8b-6729-4597-a52c-8e37a3ad27e4, pool_id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: ce05b51a-cd89-4ebb-b688-efb500dd174e, pool_id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: c01d4db6-8364-4378-8e25-1d180755f647, pool_id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: 4b4a6b84-2a5f-42e2-9e64-42ad4b741866, pool_id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: ab80f8c3-3ca5-4ec8-99ed-275ba79ffbf5, pool_id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 87286297-b93c-4248-91f6-2951af5888f5, pool_id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:451
2552024-08-29T15:14:25.285ZINFOSledAgent (StorageManager): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2562024-08-29T15:14:25.285ZINFOSledAgent (StorageManager): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2572024-08-29T15:14:25.285ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:866 request = omicron_physical_disks_ensure
2582024-08-29T15:14:25.285ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:302
2592024-08-29T15:14:25.285ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:342
2602024-08-29T15:14:25.285ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2612024-08-29T15:14:25.285ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2622024-08-29T15:14:25.441ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:360
2632024-08-29T15:14:25.441ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 05effe78-0c08-40b4-916b-cde344744ec3 (zpool)
2642024-08-29T15:14:25.702ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2652024-08-29T15:14:26.644ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2662024-08-29T15:14:26.655ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }
2672024-08-29T15:14:26.662ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:218
2682024-08-29T15:14:26.663ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:220
2692024-08-29T15:14:26.663ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:224
2702024-08-29T15:14:26.663ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:226
2712024-08-29T15:14:26.663ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2722024-08-29T15:14:26.663ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2732024-08-29T15:14:26.663ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
2742024-08-29T15:14:26.695ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2752024-08-29T15:14:26.701ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_05effe78-0c08-40b4-916b-cde344744ec3/crypt/zone file = sled-storage/src/dataset.rs:281
2762024-08-29T15:14:26.705ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
2772024-08-29T15:14:26.861ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_05effe78-0c08-40b4-916b-cde344744ec3 file = sled-storage/src/dataset.rs:367
2782024-08-29T15:14:26.868ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:440
2792024-08-29T15:14:26.868ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:357
2802024-08-29T15:14:26.868ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:342
2812024-08-29T15:14:26.868ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
2822024-08-29T15:14:26.868ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2832024-08-29T15:14:26.878ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:360
2842024-08-29T15:14:26.878ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 866d818f-82bb-444c-992a-1351dec91efa (zpool)
2852024-08-29T15:14:26.969ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }
2862024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:218
2872024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:220
2882024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:224
2892024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:226
2902024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2912024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2922024-08-29T15:14:26.977ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
2932024-08-29T15:14:27.015ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2942024-08-29T15:14:27.024ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_866d818f-82bb-444c-992a-1351dec91efa/crypt/zone file = sled-storage/src/dataset.rs:281
2952024-08-29T15:14:27.201ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_866d818f-82bb-444c-992a-1351dec91efa file = sled-storage/src/dataset.rs:367
2962024-08-29T15:14:27.209ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:440
2972024-08-29T15:14:27.209ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:357
2982024-08-29T15:14:27.209ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:342
2992024-08-29T15:14:27.209ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3002024-08-29T15:14:27.209ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3012024-08-29T15:14:27.218ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:360
3022024-08-29T15:14:27.218ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool)
3032024-08-29T15:14:27.310ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }
3042024-08-29T15:14:27.317ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:218
3052024-08-29T15:14:27.317ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:220
3062024-08-29T15:14:27.317ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:224
3072024-08-29T15:14:27.317ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:226
3082024-08-29T15:14:27.317ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3092024-08-29T15:14:27.317ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3102024-08-29T15:14:27.318ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3112024-08-29T15:14:27.352ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3122024-08-29T15:14:27.359ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_4fbbfff0-e175-4404-8cd9-acfc358b7140/crypt/zone file = sled-storage/src/dataset.rs:281
3132024-08-29T15:14:27.532ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_4fbbfff0-e175-4404-8cd9-acfc358b7140 file = sled-storage/src/dataset.rs:367
3142024-08-29T15:14:27.540ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:440
3152024-08-29T15:14:27.540ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:357
3162024-08-29T15:14:27.540ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:342
3172024-08-29T15:14:27.540ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3182024-08-29T15:14:27.540ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3192024-08-29T15:14:27.549ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:360
3202024-08-29T15:14:27.549ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool)
3212024-08-29T15:14:27.647ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }
3222024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:218
3232024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:220
3242024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:224
3252024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:226
3262024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3272024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3282024-08-29T15:14:27.655ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3292024-08-29T15:14:27.691ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3302024-08-29T15:14:27.699ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6eaf0007-0aa9-44b6-8bac-b01569f4c6a3/crypt/zone file = sled-storage/src/dataset.rs:281
3312024-08-29T15:14:27.709ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
3322024-08-29T15:14:27.872ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 file = sled-storage/src/dataset.rs:367
3332024-08-29T15:14:27.880ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:440
3342024-08-29T15:14:27.880ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:357
3352024-08-29T15:14:27.880ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:342
3362024-08-29T15:14:27.880ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3372024-08-29T15:14:27.880ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3382024-08-29T15:14:27.891ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:360
3392024-08-29T15:14:27.891ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool)
3402024-08-29T15:14:27.987ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }
3412024-08-29T15:14:27.994ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:218
3422024-08-29T15:14:27.994ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:220
3432024-08-29T15:14:27.994ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:224
3442024-08-29T15:14:27.994ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:226
3452024-08-29T15:14:27.994ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3462024-08-29T15:14:27.995ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3472024-08-29T15:14:27.995ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3482024-08-29T15:14:28.032ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3492024-08-29T15:14:28.041ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_775439b8-d6ea-4b72-9708-a30c4c7c6d6f/crypt/zone file = sled-storage/src/dataset.rs:281
3502024-08-29T15:14:28.211ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_775439b8-d6ea-4b72-9708-a30c4c7c6d6f file = sled-storage/src/dataset.rs:367
3512024-08-29T15:14:28.219ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:440
3522024-08-29T15:14:28.219ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:357
3532024-08-29T15:14:28.219ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:342
3542024-08-29T15:14:28.219ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3552024-08-29T15:14:28.219ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3562024-08-29T15:14:28.229ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:360
3572024-08-29T15:14:28.229ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = e4f77dae-534d-4b73-b38a-890d525d3747 (zpool)
3582024-08-29T15:14:28.325ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }
3592024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:218
3602024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:220
3612024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:224
3622024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:226
3632024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3642024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3652024-08-29T15:14:28.332ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3662024-08-29T15:14:28.367ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3672024-08-29T15:14:28.374ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e4f77dae-534d-4b73-b38a-890d525d3747/crypt/zone file = sled-storage/src/dataset.rs:281
3682024-08-29T15:14:28.549ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e4f77dae-534d-4b73-b38a-890d525d3747 file = sled-storage/src/dataset.rs:367
3692024-08-29T15:14:28.557ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:440
3702024-08-29T15:14:28.557ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:357
3712024-08-29T15:14:28.557ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:342
3722024-08-29T15:14:28.557ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3732024-08-29T15:14:28.557ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3742024-08-29T15:14:28.566ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:360
3752024-08-29T15:14:28.566ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool)
3762024-08-29T15:14:28.663ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External }
3772024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:218
3782024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:220
3792024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:224
3802024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:226
3812024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3822024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3832024-08-29T15:14:28.671ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
3842024-08-29T15:14:28.708ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3852024-08-29T15:14:28.711ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
3862024-08-29T15:14:28.715ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_1b55399c-9dbb-4d32-bd51-426a7996ea24/crypt/zone file = sled-storage/src/dataset.rs:281
3872024-08-29T15:14:28.887ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_1b55399c-9dbb-4d32-bd51-426a7996ea24 file = sled-storage/src/dataset.rs:367
3882024-08-29T15:14:28.895ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:440
3892024-08-29T15:14:28.895ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:357
3902024-08-29T15:14:28.895ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:342
3912024-08-29T15:14:28.895ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
3922024-08-29T15:14:28.895ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3932024-08-29T15:14:28.905ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:360
3942024-08-29T15:14:28.905ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool)
3952024-08-29T15:14:29.001ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool), kind: External }
3962024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:218
3972024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:220
3982024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:224
3992024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:226
4002024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4012024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4022024-08-29T15:14:29.010ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
4032024-08-29T15:14:29.049ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4042024-08-29T15:14:29.058ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_aad99d77-5c0c-4ae7-8cc1-ced403d6b98a/crypt/zone file = sled-storage/src/dataset.rs:281
4052024-08-29T15:14:29.233ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_aad99d77-5c0c-4ae7-8cc1-ced403d6b98a file = sled-storage/src/dataset.rs:367
4062024-08-29T15:14:29.241ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:440
4072024-08-29T15:14:29.241ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:357
4082024-08-29T15:14:29.241ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:342
4092024-08-29T15:14:29.241ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:419
4102024-08-29T15:14:29.241ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:65 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4112024-08-29T15:14:29.251ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:360
4122024-08-29T15:14:29.251ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:366 id = f4912882-4c2a-4c0d-b986-5cbff960587e (zpool)
4132024-08-29T15:14:29.346ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:174 zpool = ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External }
4142024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:218
4152024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:220
4162024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:224
4172024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:226
4182024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4192024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4202024-08-29T15:14:29.354ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:238
4212024-08-29T15:14:29.390ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4222024-08-29T15:14:29.398ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_f4912882-4c2a-4c0d-b986-5cbff960587e/crypt/zone file = sled-storage/src/dataset.rs:281
4232024-08-29T15:14:29.569ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_f4912882-4c2a-4c0d-b986-5cbff960587e file = sled-storage/src/dataset.rs:367
4242024-08-29T15:14:29.576ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:440
4252024-08-29T15:14:29.576ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:357
4262024-08-29T15:14:29.577ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4272024-08-29T15:14:29.577ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { generation: Generation(1), inner: AllDisksInner { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 5a85a3ef-48f8-4333-ae7e-46dfaa7fa523 (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6dd67a4e-189f-47cd-a6e2-0eff97293dab (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034, firmware: DiskFirmware { active_slot: 1, next_active_slot: None, slot1_read_only: true, slot_firmware_versions: [Some("synthetic 1.0")] } }, zpool_name: ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External } }))} }, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:93
4282024-08-29T15:14:29.577ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:580
4292024-08-29T15:14:29.577ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
4302024-08-29T15:14:29.577ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4312024-08-29T15:14:29.577ZINFOSledAgent: physical disks ensure: Updated storage file = sled-agent/src/sled_agent.rs:860 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4322024-08-29T15:14:29.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4332024-08-29T15:14:29.577ZINFOSledAgent: physical disks ensure: Propagating new generation of disks file = sled-agent/src/sled_agent.rs:886 generation = Generation(1) sled_id = b9c40332-a231-42db-adf0-b267d663a311
4342024-08-29T15:14:29.577ZINFOSledAgent: physical disks ensure: Updated storage monitor file = sled-agent/src/sled_agent.rs:891 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4352024-08-29T15:14:29.577ZINFOSledAgent: physical disks ensure: Updated zone bundler file = sled-agent/src/sled_agent.rs:896 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4362024-08-29T15:14:29.577ZINFOSledAgent (ProbeManager): use_only_these_disks: Processing new request file = sled-agent/src/probe_manager.rs:121 gen = Generation(1) sled_id = b9c40332-a231-42db-adf0-b267d663a311
4372024-08-29T15:14:29.577ZINFOSledAgent: physical disks ensure: Updated probes file = sled-agent/src/sled_agent.rs:902 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4382024-08-29T15:14:29.577ZINFOSledAgent (InstanceManager): use_only_these_disks: Processing new request file = sled-agent/src/instance_manager.rs:803 gen = Generation(1)
4392024-08-29T15:14:29.577ZINFOSledAgent: physical disks ensure: Updated instances file = sled-agent/src/sled_agent.rs:907 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4402024-08-29T15:14:29.577ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4292820 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:63462 req_id = 6eb7af0e-c8aa-48e8-95bb-677cb8f4cbd5 response_code = 200 uri = /omicron-physical-disks
4412024-08-29T15:14:29.609ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
4422024-08-29T15:14:29.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:62268
4432024-08-29T15:14:29.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4442024-08-29T15:14:29.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4452024-08-29T15:14:29.657ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4462024-08-29T15:14:29.657ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
4472024-08-29T15:14:29.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
4482024-08-29T15:14:29.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4492024-08-29T15:14:29.717ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4502024-08-29T15:14:29.754ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4512024-08-29T15:14:29.754ZINFOSledAgent (ServiceManager): No ledger in /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:188
4522024-08-29T15:14:29.754ZINFOSledAgent (ServiceManager): No ledger in /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:188
4532024-08-29T15:14:29.754ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4542024-08-29T15:14:29.799ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4552024-08-29T15:14:29.799ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4562024-08-29T15:14:29.799ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
4572024-08-29T15:14:29.851ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838 file = illumos-utils/src/zone.rs:316
4582024-08-29T15:14:29.855ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87 file = illumos-utils/src/zone.rs:316
4592024-08-29T15:14:29.859ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb file = illumos-utils/src/zone.rs:316
4602024-08-29T15:14:29.872ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838 file = illumos-utils/src/zone.rs:354
4612024-08-29T15:14:29.876ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87 file = illumos-utils/src/zone.rs:354
4622024-08-29T15:14:29.879ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb file = illumos-utils/src/zone.rs:354
4632024-08-29T15:14:30.719ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
4642024-08-29T15:14:31.396ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService3'/> <propval type="astring" name="gateway" value='fd00:1122:3344:1::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:1::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:1::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:1::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4652024-08-29T15:14:31.397ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:1::, width: 64 }
4662024-08-29T15:14:31.875ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService1'/> <propval type="astring" name="gateway" value='fd00:1122:3344:2::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:2::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:2::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:2::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4672024-08-29T15:14:32.351ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:2::, width: 64 }
4682024-08-29T15:14:32.351ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService2'/> <propval type="astring" name="gateway" value='fd00:1122:3344:3::2'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:3::1'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:3::1]:5353'/> <propval type="astring" name="dns_address" value='[fd00:1122:3344:3::1]:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4692024-08-29T15:14:32.351ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4702024-08-29T15:14:32.355ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:86 prefix = Ipv6Net { addr: fd00:1122:3344:3::, width: 64 }
4712024-08-29T15:14:32.355ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4722024-08-29T15:14:32.359ZINFOSledAgent (ServiceManager): Booting oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4732024-08-29T15:14:32.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 51.30061ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4742024-08-29T15:14:32.483ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 35.863528ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4752024-08-29T15:14:32.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 37.636147ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4762024-08-29T15:14:32.525ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 132.196478ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4772024-08-29T15:14:32.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 145.528795ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4782024-08-29T15:14:32.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 65.68834ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4792024-08-29T15:14:32.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 253.255053ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4802024-08-29T15:14:32.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 164.690817ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4812024-08-29T15:14:32.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 147.073015ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4822024-08-29T15:14:32.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 592.424034ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4832024-08-29T15:14:33.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 458.321473ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4842024-08-29T15:14:33.099ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 521.576482ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4852024-08-29T15:14:33.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 571.473072ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4862024-08-29T15:14:33.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 702.543989ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4872024-08-29T15:14:33.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 410.42864ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4882024-08-29T15:14:34.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 756.030551ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4892024-08-29T15:14:34.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 1.162512513s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4902024-08-29T15:14:34.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 1.405567001s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
4912024-08-29T15:14:34.821ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 1.15309281s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
4922024-08-29T15:14:35.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 1.251217681s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
4932024-08-29T15:14:35.298ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:4211
4942024-08-29T15:14:35.298ZINFOSledAgent (ServiceManager): Ensuring there is a default route file = sled-agent/src/services.rs:4221 gateway = fd00:1122:3344:101::1
4952024-08-29T15:14:35.304ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:4300
4962024-08-29T15:14:35.434ZINFOSledAgent (ServiceManager): refreshed dendrite service with new configuration file = sled-agent/src/services.rs:4348
4972024-08-29T15:14:35.434ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:4375
4982024-08-29T15:14:35.475ZINFOSledAgent (ServiceManager): refreshed lldpd service with new configuration file = sled-agent/src/services.rs:4388
4992024-08-29T15:14:35.475ZINFOSledAgent (ServiceManager): configuring MGS service file = sled-agent/src/services.rs:4252
5002024-08-29T15:14:35.527ZINFOSledAgent (ServiceManager): refreshed MGS service with new configuration file = sled-agent/src/services.rs:4294
5012024-08-29T15:14:35.527ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:4355 rack_subnet = fd00:1122:3344::
5022024-08-29T15:14:35.547ZINFOSledAgent (ServiceManager): refreshed wicketd service with new configuration file = sled-agent/src/services.rs:4366
5032024-08-29T15:14:35.547ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:4408
5042024-08-29T15:14:35.637ZINFOSledAgent (ServiceManager): refreshed mgd service with new configuration file = sled-agent/src/services.rs:4440
5052024-08-29T15:14:35.637ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4446
5062024-08-29T15:14:35.678ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 1.199170507s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
5072024-08-29T15:14:35.748ZINFOSledAgent (ServiceManager): refreshed mg-ddm service with new configuration file = sled-agent/src/services.rs:4487
5082024-08-29T15:14:35.748ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:366
5092024-08-29T15:14:35.781ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:571 sled_id = b9c40332-a231-42db-adf0-b267d663a311
5102024-08-29T15:14:35.781ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 29.63559ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5112024-08-29T15:14:35.812ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 52.567226ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5122024-08-29T15:14:35.866ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 126.206105ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5132024-08-29T15:14:35.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 598.019722ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
5142024-08-29T15:14:35.993ZWARNSledAgent (ServiceManager): Failed to get switch ID from MGS (retrying in 330.493544ms) error = Communication Error: error sending request for url (http://[fd00:1122:3344:101::2]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/bootstrap/early_networking.rs:385
5152024-08-29T15:14:36.325ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:414
5162024-08-29T15:14:36.335ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::2]:54464
5172024-08-29T15:14:36.336ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::2]:54464 req_id = 3f85cc38-20af-4573-959e-5b0614214a63 response_code = 200 uri = /sled-identifiers
5182024-08-29T15:14:36.358ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:676
5192024-08-29T15:14:36.358ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:714
5202024-08-29T15:14:36.358ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:717 uptime = 0
5212024-08-29T15:14:36.358ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:436
5222024-08-29T15:14:36.396ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
5232024-08-29T15:14:36.444ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:451
5242024-08-29T15:14:36.444ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5252024-08-29T15:14:36.444ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-08-29T15:14:36.444ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5272024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5292024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5312024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5332024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5352024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5372024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5392024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5412024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5432024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:529 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5452024-08-29T15:14:36.445ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:550 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-08-29T15:14:36.492ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838") failed: Property not found. retry in 1.474559347s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
5472024-08-29T15:14:36.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 647.402593ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
5482024-08-29T15:14:36.887ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87") failed: Property not found. retry in 1.257920072s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
5492024-08-29T15:14:37.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb") failed: Property not found. retry in 1.431049397s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
5502024-08-29T15:14:38.696ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_internal_dns_ccbc07cd-fbf0-43da-b57b-17b3debd7838
5512024-08-29T15:14:38.696ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_internal_dns_c7342971-675e-4b24-995a-35efbafcac87
5522024-08-29T15:14:38.696ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_internal_dns_285489bf-dd9f-4685-a3f1-46733c794cfb
5532024-08-29T15:14:38.696ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5542024-08-29T15:14:38.696ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5552024-08-29T15:14:38.697ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 9087386 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:62268 req_id = 99f5e8a2-10f8-47ee-b88a-59c7901d0cb0 response_code = 204 uri = /omicron-zones
5562024-08-29T15:14:38.699ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5572024-08-29T15:14:38.748ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5582024-08-29T15:14:38.796ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:555
5592024-08-29T15:14:38.844ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:591
5602024-08-29T15:14:38.844ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5612024-08-29T15:14:38.877ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5622024-08-29T15:14:38.878ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5632024-08-29T15:14:38.878ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5642024-08-29T15:14:38.907ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
5652024-08-29T15:14:38.907ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65086
5662024-08-29T15:14:38.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5672024-08-29T15:14:38.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5682024-08-29T15:14:38.929ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5692024-08-29T15:14:38.929ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
5702024-08-29T15:14:38.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
5712024-08-29T15:14:38.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5722024-08-29T15:14:38.963ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5732024-08-29T15:14:38.964ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
5742024-08-29T15:14:38.964ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
5752024-08-29T15:14:38.964ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5762024-08-29T15:14:38.975ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
5772024-08-29T15:14:39.008ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
5782024-08-29T15:14:39.008ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
5792024-08-29T15:14:39.008ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
5802024-08-29T15:14:39.039ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 223, 79]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 } }) }
5812024-08-29T15:14:39.039ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = boundary_ntp
5822024-08-29T15:14:39.039ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:83
5832024-08-29T15:14:39.039ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:90
5842024-08-29T15:14:39.040ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.169 file = clients/dpd-client/src/lib.rs:118
5852024-08-29T15:14:39.040ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
5862024-08-29T15:14:39.064ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25 file = illumos-utils/src/zone.rs:316
5872024-08-29T15:14:39.081ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25 file = illumos-utils/src/zone.rs:354
5882024-08-29T15:14:39.701ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService4'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value='fd00:1122:3344:100::/56'/> <propval type="boolean" name="boundary" value='true'/> <propval type="astring" name="boundary_pool" value='boundary-ntp.control-plane.oxide.internal'/> <propval type="astring" name="server" value='0.pool.ntp.org'/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte0'/> <propval type="astring" name="gateway" value='172.30.3.1'/> <propval type="astring" name="ip" value='172.30.3.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5892024-08-29T15:14:39.701ZINFOSledAgent (ServiceManager): Booting oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5902024-08-29T15:14:39.834ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 41.884875ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5912024-08-29T15:14:39.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 93.495881ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5922024-08-29T15:14:40.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 146.390482ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5932024-08-29T15:14:40.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 417.468809ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5942024-08-29T15:14:40.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 475.517419ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5952024-08-29T15:14:41.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 1.128076891s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5962024-08-29T15:14:42.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 1.124862757s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5972024-08-29T15:14:43.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 925.492782ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5982024-08-29T15:14:44.280ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25") failed: Property not found. retry in 641.688766ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
5992024-08-29T15:14:44.941ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25
6002024-08-29T15:14:44.942ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6012024-08-29T15:14:44.942ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6022024-08-29T15:14:44.942ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6035064 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:65086 req_id = cbb35c06-0afd-4bef-8a71-ed9294653692 response_code = 204 uri = /omicron-zones
6032024-08-29T15:14:44.945ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:632
6042024-08-29T15:14:44.976ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6052024-08-29T15:14:44.977ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65029
6062024-08-29T15:14:44.985ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6072024-08-29T15:14:44.985ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8307 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:65029 req_id = 1e8f2fcb-2070-4802-9805-e489ee5ef05f response_code = 500 uri = /timesync
6082024-08-29T15:14:44.986ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6092024-08-29T15:14:45.073ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6102024-08-29T15:14:45.073ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59692
6112024-08-29T15:14:47.081ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6122024-08-29T15:14:47.082ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 2008023 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59692 req_id = ec8942c4-ec59-49a5-b5e0-8aec4f74d84f response_code = 500 uri = /timesync
6132024-08-29T15:14:47.085ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6142024-08-29T15:14:47.264ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6152024-08-29T15:14:47.264ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:45972
6162024-08-29T15:14:47.272ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6172024-08-29T15:14:47.273ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8640 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:45972 req_id = ecea9fe4-5884-4061-9da2-8f23b4936678 response_code = 500 uri = /timesync
6182024-08-29T15:14:47.276ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6192024-08-29T15:14:47.516ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6202024-08-29T15:14:47.516ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37823
6212024-08-29T15:14:47.524ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6222024-08-29T15:14:47.524ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 7587 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37823 req_id = 05934969-4c70-4395-b95b-569140a4a9e4 response_code = 500 uri = /timesync
6232024-08-29T15:14:47.525ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6242024-08-29T15:14:47.789ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6252024-08-29T15:14:47.789ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:37245
6262024-08-29T15:14:48.798ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6272024-08-29T15:14:48.798ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 1009178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:37245 req_id = 27d7e1a0-782c-4ba1-a07a-1b6c103a4f8e response_code = 500 uri = /timesync
6282024-08-29T15:14:48.801ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6292024-08-29T15:14:49.022ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6302024-08-29T15:14:49.022ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33191
6312024-08-29T15:14:49.030ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6322024-08-29T15:14:49.030ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8503 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33191 req_id = 4f898ee9-e2b6-4181-a801-4a42cc03fb80 response_code = 500 uri = /timesync
6332024-08-29T15:14:49.032ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6342024-08-29T15:14:49.296ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6352024-08-29T15:14:49.296ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57885
6362024-08-29T15:14:49.305ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6372024-08-29T15:14:49.305ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8219 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:57885 req_id = c1866df3-b955-4c26-8f63-026f5a143b81 response_code = 500 uri = /timesync
6382024-08-29T15:14:49.307ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6392024-08-29T15:14:49.547ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6402024-08-29T15:14:49.547ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:54855
6412024-08-29T15:14:50.556ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6422024-08-29T15:14:50.556ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 1009005 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:54855 req_id = a01e0f37-f257-4d89-8502-ec17a9136527 response_code = 500 uri = /timesync
6432024-08-29T15:14:50.559ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6442024-08-29T15:14:50.941ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6452024-08-29T15:14:50.941ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:55155
6462024-08-29T15:14:50.949ZERROSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_43a6fa50-4493-4748-921d-7ff7c23c0e25': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3852
6472024-08-29T15:14:50.950ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 8097 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:55155 req_id = 6293a0e2-4e78-466d-b7be-54851340a233 response_code = 500 uri = /timesync
6482024-08-29T15:14:50.951ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6492024-08-29T15:14:51.371ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6502024-08-29T15:14:51.371ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49790
6512024-08-29T15:14:51.380ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 8909 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49790 req_id = aac1c1dc-5d71-4d8e-8b78-d209d485fa8e response_code = 200 uri = /timesync
6522024-08-29T15:14:51.382ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6532024-08-29T15:14:51.382ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6542024-08-29T15:14:52.030ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6552024-08-29T15:14:52.030ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50853
6562024-08-29T15:14:52.038ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7913 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50853 req_id = a62b2d77-36ee-47ba-a16b-fd1d217abc93 response_code = 200 uri = /timesync
6572024-08-29T15:14:52.041ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6582024-08-29T15:14:52.041ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6592024-08-29T15:14:52.439ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6602024-08-29T15:14:52.439ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:59328
6612024-08-29T15:14:52.447ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7854 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:59328 req_id = 1092fd6c-177b-4c73-9244-666e4205f821 response_code = 200 uri = /timesync
6622024-08-29T15:14:52.450ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6632024-08-29T15:14:52.450ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:662
6642024-08-29T15:14:53.050ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:612
6652024-08-29T15:14:53.050ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:43296
6662024-08-29T15:14:53.071ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 20706 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:43296 req_id = c73c190d-ef62-4c50-aeeb-3e924a3031d1 response_code = 200 uri = /timesync
6672024-08-29T15:14:53.074ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2616771612, ip_addr: 155.248.196.28, stratum: 3, ref_time: 1724944492.8194437, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:640
6682024-08-29T15:14:53.074ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1266
6692024-08-29T15:14:53.106ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6702024-08-29T15:14:53.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41420
6712024-08-29T15:14:53.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6722024-08-29T15:14:53.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6732024-08-29T15:14:53.127ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6742024-08-29T15:14:53.127ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
6752024-08-29T15:14:53.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
6762024-08-29T15:14:53.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6772024-08-29T15:14:53.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6782024-08-29T15:14:53.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6792024-08-29T15:14:53.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6802024-08-29T15:14:53.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
6812024-08-29T15:14:53.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6822024-08-29T15:14:53.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6832024-08-29T15:14:53.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6842024-08-29T15:14:53.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
6852024-08-29T15:14:53.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
6862024-08-29T15:14:53.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
6872024-08-29T15:14:53.426ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6882024-08-29T15:14:53.426ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
6892024-08-29T15:14:53.426ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
6902024-08-29T15:14:53.426ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
6912024-08-29T15:14:53.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
6922024-08-29T15:14:53.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
6932024-08-29T15:14:53.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
6942024-08-29T15:14:53.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
6952024-08-29T15:14:53.458ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
6962024-08-29T15:14:53.458ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 351910 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 85c8d87d-d078-4e2b-91ea-ae7392273da6 response_code = 500 uri = /omicron-zones
6972024-08-29T15:14:53.458ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "85c8d87d-d078-4e2b-91ea-ae7392273da6", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "85c8d87d-d078-4e2b-91ea-ae7392273da6" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 145.186357ms sled_address = [fd00:1122:3344:101::1]:12345
6982024-08-29T15:14:53.605ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
6992024-08-29T15:14:53.606ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7002024-08-29T15:14:53.606ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7012024-08-29T15:14:53.624ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7022024-08-29T15:14:53.624ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
7032024-08-29T15:14:53.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7042024-08-29T15:14:53.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7052024-08-29T15:14:53.660ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7062024-08-29T15:14:53.660ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7072024-08-29T15:14:53.678ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7082024-08-29T15:14:53.678ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
7092024-08-29T15:14:53.697ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7102024-08-29T15:14:53.697ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7112024-08-29T15:14:53.715ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7122024-08-29T15:14:53.715ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7132024-08-29T15:14:53.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7142024-08-29T15:14:53.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7152024-08-29T15:14:53.752ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7162024-08-29T15:14:53.752ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
7172024-08-29T15:14:53.753ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
7182024-08-29T15:14:53.753ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7192024-08-29T15:14:53.784ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
7202024-08-29T15:14:53.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
7212024-08-29T15:14:53.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
7222024-08-29T15:14:53.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
7232024-08-29T15:14:53.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
7242024-08-29T15:14:53.785ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 179941 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 9d447e89-6706-49dc-8d74-5cfaba5934d7 response_code = 500 uri = /omicron-zones
7252024-08-29T15:14:53.785ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9d447e89-6706-49dc-8d74-5cfaba5934d7", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9d447e89-6706-49dc-8d74-5cfaba5934d7" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 85.08457ms sled_address = [fd00:1122:3344:101::1]:12345
7262024-08-29T15:14:53.871ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7272024-08-29T15:14:53.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7282024-08-29T15:14:53.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7292024-08-29T15:14:53.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7302024-08-29T15:14:53.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
7312024-08-29T15:14:53.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7322024-08-29T15:14:53.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7332024-08-29T15:14:53.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7342024-08-29T15:14:53.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7352024-08-29T15:14:53.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7362024-08-29T15:14:53.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
7372024-08-29T15:14:53.964ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7382024-08-29T15:14:53.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7392024-08-29T15:14:53.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7402024-08-29T15:14:53.982ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7412024-08-29T15:14:54.000ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7422024-08-29T15:14:54.000ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7432024-08-29T15:14:54.018ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7442024-08-29T15:14:54.018ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
7452024-08-29T15:14:54.018ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
7462024-08-29T15:14:54.018ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7472024-08-29T15:14:54.050ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
7482024-08-29T15:14:54.050ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
7492024-08-29T15:14:54.050ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
7502024-08-29T15:14:54.050ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
7512024-08-29T15:14:54.050ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
7522024-08-29T15:14:54.050ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 179023 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = c1c2ba53-6206-4a53-a4d9-a54162899729 response_code = 500 uri = /omicron-zones
7532024-08-29T15:14:54.050ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c1c2ba53-6206-4a53-a4d9-a54162899729", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:53 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c1c2ba53-6206-4a53-a4d9-a54162899729" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 141.594144ms sled_address = [fd00:1122:3344:101::1]:12345
7542024-08-29T15:14:54.066ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7552024-08-29T15:14:54.067ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7562024-08-29T15:14:54.067ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7572024-08-29T15:14:54.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7582024-08-29T15:14:54.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
7592024-08-29T15:14:54.103ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7602024-08-29T15:14:54.103ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7612024-08-29T15:14:54.122ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7622024-08-29T15:14:54.122ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7632024-08-29T15:14:54.140ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7642024-08-29T15:14:54.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
7652024-08-29T15:14:54.159ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7662024-08-29T15:14:54.159ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7672024-08-29T15:14:54.177ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7682024-08-29T15:14:54.177ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7692024-08-29T15:14:54.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7702024-08-29T15:14:54.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7712024-08-29T15:14:54.215ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7722024-08-29T15:14:54.215ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
7732024-08-29T15:14:54.215ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
7742024-08-29T15:14:54.215ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
7752024-08-29T15:14:54.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
7762024-08-29T15:14:54.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
7772024-08-29T15:14:54.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
7782024-08-29T15:14:54.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
7792024-08-29T15:14:54.248ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
7802024-08-29T15:14:54.248ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181957 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = f862700a-56ee-4b76-8fb8-dc79491b87e0 response_code = 500 uri = /omicron-zones
7812024-08-29T15:14:54.248ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f862700a-56ee-4b76-8fb8-dc79491b87e0", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f862700a-56ee-4b76-8fb8-dc79491b87e0" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 190.262652ms sled_address = [fd00:1122:3344:101::1]:12345
7822024-08-29T15:14:54.438ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
7832024-08-29T15:14:54.439ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7842024-08-29T15:14:54.439ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7852024-08-29T15:14:54.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7862024-08-29T15:14:54.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
7872024-08-29T15:14:54.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
7882024-08-29T15:14:54.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
7892024-08-29T15:14:54.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7902024-08-29T15:14:54.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7912024-08-29T15:14:54.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7922024-08-29T15:14:54.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
7932024-08-29T15:14:54.529ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7942024-08-29T15:14:54.529ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7952024-08-29T15:14:54.547ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7962024-08-29T15:14:54.547ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
7972024-08-29T15:14:54.565ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
7982024-08-29T15:14:54.565ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
7992024-08-29T15:14:54.584ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8002024-08-29T15:14:54.584ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
8012024-08-29T15:14:54.585ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
8022024-08-29T15:14:54.585ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8032024-08-29T15:14:54.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
8042024-08-29T15:14:54.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
8052024-08-29T15:14:54.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
8062024-08-29T15:14:54.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
8072024-08-29T15:14:54.616ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
8082024-08-29T15:14:54.616ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 177855 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = f974c56c-db90-4cc4-b1d7-f15c6abfbc37 response_code = 500 uri = /omicron-zones
8092024-08-29T15:14:54.617ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f974c56c-db90-4cc4-b1d7-f15c6abfbc37", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f974c56c-db90-4cc4-b1d7-f15c6abfbc37" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 147.416271ms sled_address = [fd00:1122:3344:101::1]:12345
8102024-08-29T15:14:54.765ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8112024-08-29T15:14:54.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8122024-08-29T15:14:54.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8132024-08-29T15:14:54.783ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8142024-08-29T15:14:54.783ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
8152024-08-29T15:14:54.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8162024-08-29T15:14:54.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8172024-08-29T15:14:54.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8182024-08-29T15:14:54.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8192024-08-29T15:14:54.837ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8202024-08-29T15:14:54.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
8212024-08-29T15:14:54.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8222024-08-29T15:14:54.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8232024-08-29T15:14:54.873ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8242024-08-29T15:14:54.873ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8252024-08-29T15:14:54.892ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8262024-08-29T15:14:54.892ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8272024-08-29T15:14:54.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8282024-08-29T15:14:54.909ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
8292024-08-29T15:14:54.909ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
8302024-08-29T15:14:54.909ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8312024-08-29T15:14:54.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
8322024-08-29T15:14:54.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
8332024-08-29T15:14:54.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
8342024-08-29T15:14:54.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
8352024-08-29T15:14:54.943ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
8362024-08-29T15:14:54.943ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 177890 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = de5a95ab-1c97-4a63-92c2-9511b6324c1e response_code = 500 uri = /omicron-zones
8372024-08-29T15:14:54.943ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "de5a95ab-1c97-4a63-92c2-9511b6324c1e", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "de5a95ab-1c97-4a63-92c2-9511b6324c1e" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 332.283817ms sled_address = [fd00:1122:3344:101::1]:12345
8382024-08-29T15:14:55.277ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8392024-08-29T15:14:55.278ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8402024-08-29T15:14:55.278ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8412024-08-29T15:14:55.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8422024-08-29T15:14:55.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
8432024-08-29T15:14:55.314ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8442024-08-29T15:14:55.314ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8452024-08-29T15:14:55.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8462024-08-29T15:14:55.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8472024-08-29T15:14:55.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8482024-08-29T15:14:55.351ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
8492024-08-29T15:14:55.370ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8502024-08-29T15:14:55.370ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8512024-08-29T15:14:55.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8522024-08-29T15:14:55.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8532024-08-29T15:14:55.404ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 128.036551ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
8542024-08-29T15:14:55.406ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8552024-08-29T15:14:55.406ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8562024-08-29T15:14:55.424ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8572024-08-29T15:14:55.424ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
8582024-08-29T15:14:55.424ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
8592024-08-29T15:14:55.424ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8602024-08-29T15:14:55.456ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
8612024-08-29T15:14:55.456ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
8622024-08-29T15:14:55.456ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
8632024-08-29T15:14:55.456ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
8642024-08-29T15:14:55.456ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
8652024-08-29T15:14:55.456ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 177825 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 04ad6c4f-9818-4235-82f9-0cce29b1bf2b response_code = 500 uri = /omicron-zones
8662024-08-29T15:14:55.456ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "04ad6c4f-9818-4235-82f9-0cce29b1bf2b", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "04ad6c4f-9818-4235-82f9-0cce29b1bf2b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 434.328132ms sled_address = [fd00:1122:3344:101::1]:12345
8672024-08-29T15:14:55.893ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8682024-08-29T15:14:55.893ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8692024-08-29T15:14:55.893ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8702024-08-29T15:14:55.914ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8712024-08-29T15:14:55.914ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
8722024-08-29T15:14:55.933ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8732024-08-29T15:14:55.933ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8742024-08-29T15:14:55.952ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8752024-08-29T15:14:55.952ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8762024-08-29T15:14:55.971ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8772024-08-29T15:14:55.971ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
8782024-08-29T15:14:55.989ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8792024-08-29T15:14:55.989ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8802024-08-29T15:14:56.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8812024-08-29T15:14:56.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
8822024-08-29T15:14:56.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
8832024-08-29T15:14:56.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
8842024-08-29T15:14:56.045ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8852024-08-29T15:14:56.046ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
8862024-08-29T15:14:56.046ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
8872024-08-29T15:14:56.046ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
8882024-08-29T15:14:56.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
8892024-08-29T15:14:56.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
8902024-08-29T15:14:56.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
8912024-08-29T15:14:56.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
8922024-08-29T15:14:56.078ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
8932024-08-29T15:14:56.079ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185327 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 98bb2684-7f4a-4f8e-982f-8f2fbad817d3 response_code = 500 uri = /omicron-zones
8942024-08-29T15:14:56.079ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "98bb2684-7f4a-4f8e-982f-8f2fbad817d3", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:55 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "98bb2684-7f4a-4f8e-982f-8f2fbad817d3" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 232.167435ms sled_address = [fd00:1122:3344:101::1]:12345
8952024-08-29T15:14:56.313ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
8962024-08-29T15:14:56.313ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8972024-08-29T15:14:56.313ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8982024-08-29T15:14:56.333ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
8992024-08-29T15:14:56.333ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
9002024-08-29T15:14:56.351ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9012024-08-29T15:14:56.351ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9022024-08-29T15:14:56.370ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9032024-08-29T15:14:56.370ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9042024-08-29T15:14:56.388ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9052024-08-29T15:14:56.388ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
9062024-08-29T15:14:56.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9072024-08-29T15:14:56.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9082024-08-29T15:14:56.423ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9092024-08-29T15:14:56.423ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9102024-08-29T15:14:56.441ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9112024-08-29T15:14:56.441ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9122024-08-29T15:14:56.459ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9132024-08-29T15:14:56.459ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
9142024-08-29T15:14:56.459ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
9152024-08-29T15:14:56.459ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9162024-08-29T15:14:56.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
9172024-08-29T15:14:56.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
9182024-08-29T15:14:56.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
9192024-08-29T15:14:56.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
9202024-08-29T15:14:56.492ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
9212024-08-29T15:14:56.492ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 178814 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = cf49dc43-9af9-4048-9c0c-335befd6f628 response_code = 500 uri = /omicron-zones
9222024-08-29T15:14:56.492ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "cf49dc43-9af9-4048-9c0c-335befd6f628", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "cf49dc43-9af9-4048-9c0c-335befd6f628" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 581.039957ms sled_address = [fd00:1122:3344:101::1]:12345
9232024-08-29T15:14:57.076ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9242024-08-29T15:14:57.076ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9252024-08-29T15:14:57.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9262024-08-29T15:14:57.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9272024-08-29T15:14:57.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
9282024-08-29T15:14:57.113ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9292024-08-29T15:14:57.113ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9302024-08-29T15:14:57.131ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9312024-08-29T15:14:57.131ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9322024-08-29T15:14:57.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9332024-08-29T15:14:57.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
9342024-08-29T15:14:57.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9352024-08-29T15:14:57.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9362024-08-29T15:14:57.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9372024-08-29T15:14:57.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9382024-08-29T15:14:57.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9392024-08-29T15:14:57.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9402024-08-29T15:14:57.226ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9412024-08-29T15:14:57.226ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
9422024-08-29T15:14:57.226ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
9432024-08-29T15:14:57.226ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9442024-08-29T15:14:57.258ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
9452024-08-29T15:14:57.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
9462024-08-29T15:14:57.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
9472024-08-29T15:14:57.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
9482024-08-29T15:14:57.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
9492024-08-29T15:14:57.259ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 182042 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 3ee439aa-ce1c-40af-8f47-ecae218d7635 response_code = 500 uri = /omicron-zones
9502024-08-29T15:14:57.259ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "3ee439aa-ce1c-40af-8f47-ecae218d7635", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:56 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "3ee439aa-ce1c-40af-8f47-ecae218d7635" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 325.931023ms sled_address = [fd00:1122:3344:101::1]:12345
9512024-08-29T15:14:57.587ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9522024-08-29T15:14:57.588ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9532024-08-29T15:14:57.588ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9542024-08-29T15:14:57.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9552024-08-29T15:14:57.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
9562024-08-29T15:14:57.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9572024-08-29T15:14:57.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9582024-08-29T15:14:57.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9592024-08-29T15:14:57.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9602024-08-29T15:14:57.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9612024-08-29T15:14:57.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
9622024-08-29T15:14:57.682ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9632024-08-29T15:14:57.682ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9642024-08-29T15:14:57.702ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9652024-08-29T15:14:57.702ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9662024-08-29T15:14:57.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9672024-08-29T15:14:57.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9682024-08-29T15:14:57.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9692024-08-29T15:14:57.740ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
9702024-08-29T15:14:57.740ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
9712024-08-29T15:14:57.740ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9722024-08-29T15:14:57.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
9732024-08-29T15:14:57.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
9742024-08-29T15:14:57.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
9752024-08-29T15:14:57.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
9762024-08-29T15:14:57.773ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
9772024-08-29T15:14:57.773ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185487 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 51efeef9-b20e-44b3-b928-022c171f1a64 response_code = 500 uri = /omicron-zones
9782024-08-29T15:14:57.773ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "51efeef9-b20e-44b3-b928-022c171f1a64", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:57 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "51efeef9-b20e-44b3-b928-022c171f1a64" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 810.891211ms sled_address = [fd00:1122:3344:101::1]:12345
9792024-08-29T15:14:58.587ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
9802024-08-29T15:14:58.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9812024-08-29T15:14:58.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9822024-08-29T15:14:58.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9832024-08-29T15:14:58.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
9842024-08-29T15:14:58.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
9852024-08-29T15:14:58.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9862024-08-29T15:14:58.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9872024-08-29T15:14:58.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9882024-08-29T15:14:58.666ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9892024-08-29T15:14:58.666ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
9902024-08-29T15:14:58.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9912024-08-29T15:14:58.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9922024-08-29T15:14:58.701ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9932024-08-29T15:14:58.701ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
9942024-08-29T15:14:58.721ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
9952024-08-29T15:14:58.721ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
9962024-08-29T15:14:58.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
9972024-08-29T15:14:58.739ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
9982024-08-29T15:14:58.739ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
9992024-08-29T15:14:58.739ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10002024-08-29T15:14:58.771ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
10012024-08-29T15:14:58.771ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
10022024-08-29T15:14:58.771ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
10032024-08-29T15:14:58.771ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
10042024-08-29T15:14:58.771ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
10052024-08-29T15:14:58.771ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183421 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = b02513c6-22fa-46e3-872d-a0a1d49f4498 response_code = 500 uri = /omicron-zones
10062024-08-29T15:14:58.771ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b02513c6-22fa-46e3-872d-a0a1d49f4498", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b02513c6-22fa-46e3-872d-a0a1d49f4498" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 791.599088ms sled_address = [fd00:1122:3344:101::1]:12345
10072024-08-29T15:14:59.566ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10082024-08-29T15:14:59.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10092024-08-29T15:14:59.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10102024-08-29T15:14:59.586ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10112024-08-29T15:14:59.586ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
10122024-08-29T15:14:59.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10132024-08-29T15:14:59.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10142024-08-29T15:14:59.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10152024-08-29T15:14:59.623ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10162024-08-29T15:14:59.642ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10172024-08-29T15:14:59.642ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
10182024-08-29T15:14:59.661ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10192024-08-29T15:14:59.661ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10202024-08-29T15:14:59.680ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10212024-08-29T15:14:59.680ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10222024-08-29T15:14:59.698ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10232024-08-29T15:14:59.698ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10242024-08-29T15:14:59.715ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10252024-08-29T15:14:59.715ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
10262024-08-29T15:14:59.715ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
10272024-08-29T15:14:59.715ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10282024-08-29T15:14:59.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
10292024-08-29T15:14:59.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
10302024-08-29T15:14:59.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
10312024-08-29T15:14:59.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
10322024-08-29T15:14:59.748ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
10332024-08-29T15:14:59.748ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181609 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 2af92ffd-a89f-4fe7-92bc-6bcef4a052a3 response_code = 500 uri = /omicron-zones
10342024-08-29T15:14:59.748ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2af92ffd-a89f-4fe7-92bc-6bcef4a052a3", "content-length": "124", "date": "Thu, 29 Aug 2024 15:14:59 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2af92ffd-a89f-4fe7-92bc-6bcef4a052a3" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.174093048s sled_address = [fd00:1122:3344:101::1]:12345
10352024-08-29T15:15:00.927ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10362024-08-29T15:15:00.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10372024-08-29T15:15:00.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10382024-08-29T15:15:00.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10392024-08-29T15:15:00.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
10402024-08-29T15:15:00.976ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10412024-08-29T15:15:00.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10422024-08-29T15:15:00.998ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10432024-08-29T15:15:00.998ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10442024-08-29T15:15:01.020ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10452024-08-29T15:15:01.020ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
10462024-08-29T15:15:01.043ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10472024-08-29T15:15:01.043ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10482024-08-29T15:15:01.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10492024-08-29T15:15:01.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10502024-08-29T15:15:01.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10512024-08-29T15:15:01.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10522024-08-29T15:15:01.109ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10532024-08-29T15:15:01.109ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
10542024-08-29T15:15:01.109ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
10552024-08-29T15:15:01.110ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10562024-08-29T15:15:01.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
10572024-08-29T15:15:01.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
10582024-08-29T15:15:01.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
10592024-08-29T15:15:01.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
10602024-08-29T15:15:01.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
10612024-08-29T15:15:01.144ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 215575 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 699a7b0e-4d07-432b-8f95-86d7831b8a8f response_code = 500 uri = /omicron-zones
10622024-08-29T15:15:01.144ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "699a7b0e-4d07-432b-8f95-86d7831b8a8f", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:00 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "699a7b0e-4d07-432b-8f95-86d7831b8a8f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.078472214s sled_address = [fd00:1122:3344:101::1]:12345
10632024-08-29T15:15:02.227ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10642024-08-29T15:15:02.227ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10652024-08-29T15:15:02.227ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10662024-08-29T15:15:02.246ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10672024-08-29T15:15:02.246ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
10682024-08-29T15:15:02.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10692024-08-29T15:15:02.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10702024-08-29T15:15:02.283ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10712024-08-29T15:15:02.283ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10722024-08-29T15:15:02.301ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10732024-08-29T15:15:02.302ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
10742024-08-29T15:15:02.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10752024-08-29T15:15:02.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10762024-08-29T15:15:02.340ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10772024-08-29T15:15:02.340ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
10782024-08-29T15:15:02.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10792024-08-29T15:15:02.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
10802024-08-29T15:15:02.378ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10812024-08-29T15:15:02.378ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
10822024-08-29T15:15:02.378ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
10832024-08-29T15:15:02.379ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
10842024-08-29T15:15:02.411ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
10852024-08-29T15:15:02.411ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
10862024-08-29T15:15:02.411ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
10872024-08-29T15:15:02.411ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
10882024-08-29T15:15:02.411ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
10892024-08-29T15:15:02.411ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 0303b6c4-de46-4efc-8435-52f5495344f5 response_code = 500 uri = /omicron-zones
10902024-08-29T15:15:02.412ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "0303b6c4-de46-4efc-8435-52f5495344f5", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:02 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "0303b6c4-de46-4efc-8435-52f5495344f5" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.305149646s sled_address = [fd00:1122:3344:101::1]:12345
10912024-08-29T15:15:03.722ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
10922024-08-29T15:15:03.722ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10932024-08-29T15:15:03.722ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10942024-08-29T15:15:03.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10952024-08-29T15:15:03.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
10962024-08-29T15:15:03.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
10972024-08-29T15:15:03.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10982024-08-29T15:15:03.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
10992024-08-29T15:15:03.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11002024-08-29T15:15:03.801ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11012024-08-29T15:15:03.801ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
11022024-08-29T15:15:03.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11032024-08-29T15:15:03.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11042024-08-29T15:15:03.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11052024-08-29T15:15:03.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11062024-08-29T15:15:03.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11072024-08-29T15:15:03.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11082024-08-29T15:15:03.875ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11092024-08-29T15:15:03.875ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
11102024-08-29T15:15:03.875ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
11112024-08-29T15:15:03.875ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11122024-08-29T15:15:03.908ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
11132024-08-29T15:15:03.908ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
11142024-08-29T15:15:03.908ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
11152024-08-29T15:15:03.908ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
11162024-08-29T15:15:03.908ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
11172024-08-29T15:15:03.908ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185641 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 2698c763-101e-41c6-bfa7-f0039a78dc57 response_code = 500 uri = /omicron-zones
11182024-08-29T15:15:03.908ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2698c763-101e-41c6-bfa7-f0039a78dc57", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2698c763-101e-41c6-bfa7-f0039a78dc57" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 1.299137698s sled_address = [fd00:1122:3344:101::1]:12345
11192024-08-29T15:15:05.002ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50006
11202024-08-29T15:15:05.002ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 53 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50006 req_id = db8f9734-c022-4d98-bb90-d9034fd7d4ed response_code = 200 uri = /baseboard
11212024-08-29T15:15:05.213ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11222024-08-29T15:15:05.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11232024-08-29T15:15:05.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11242024-08-29T15:15:05.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11252024-08-29T15:15:05.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
11262024-08-29T15:15:05.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11272024-08-29T15:15:05.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11282024-08-29T15:15:05.274ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11292024-08-29T15:15:05.274ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11302024-08-29T15:15:05.292ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11312024-08-29T15:15:05.292ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
11322024-08-29T15:15:05.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11332024-08-29T15:15:05.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11342024-08-29T15:15:05.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11352024-08-29T15:15:05.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11362024-08-29T15:15:05.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11372024-08-29T15:15:05.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11382024-08-29T15:15:05.367ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11392024-08-29T15:15:05.367ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
11402024-08-29T15:15:05.367ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
11412024-08-29T15:15:05.367ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11422024-08-29T15:15:05.400ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
11432024-08-29T15:15:05.400ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
11442024-08-29T15:15:05.400ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
11452024-08-29T15:15:05.400ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
11462024-08-29T15:15:05.400ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
11472024-08-29T15:15:05.400ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186860 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = afcba35c-51f3-48a8-b2a0-6a1760ef2a8b response_code = 500 uri = /omicron-zones
11482024-08-29T15:15:05.400ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "afcba35c-51f3-48a8-b2a0-6a1760ef2a8b", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "afcba35c-51f3-48a8-b2a0-6a1760ef2a8b" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.555828637s sled_address = [fd00:1122:3344:101::1]:12345
11492024-08-29T15:15:07.964ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11502024-08-29T15:15:07.964ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11512024-08-29T15:15:07.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11522024-08-29T15:15:07.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11532024-08-29T15:15:07.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
11542024-08-29T15:15:08.004ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11552024-08-29T15:15:08.004ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11562024-08-29T15:15:08.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11572024-08-29T15:15:08.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11582024-08-29T15:15:08.044ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11592024-08-29T15:15:08.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
11602024-08-29T15:15:08.064ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11612024-08-29T15:15:08.064ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11622024-08-29T15:15:08.082ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11632024-08-29T15:15:08.083ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11642024-08-29T15:15:08.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11652024-08-29T15:15:08.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11662024-08-29T15:15:08.120ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11672024-08-29T15:15:08.120ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
11682024-08-29T15:15:08.120ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
11692024-08-29T15:15:08.120ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11702024-08-29T15:15:08.151ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
11712024-08-29T15:15:08.151ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
11722024-08-29T15:15:08.151ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
11732024-08-29T15:15:08.151ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
11742024-08-29T15:15:08.151ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
11752024-08-29T15:15:08.151ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186909 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = e957539f-e95f-4d49-a9b1-27db29bbb93a response_code = 500 uri = /omicron-zones
11762024-08-29T15:15:08.151ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e957539f-e95f-4d49-a9b1-27db29bbb93a", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e957539f-e95f-4d49-a9b1-27db29bbb93a" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.781391694s sled_address = [fd00:1122:3344:101::1]:12345
11772024-08-29T15:15:10.609ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 748.301471ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
11782024-08-29T15:15:10.941ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
11792024-08-29T15:15:10.941ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11802024-08-29T15:15:10.941ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11812024-08-29T15:15:10.962ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11822024-08-29T15:15:10.962ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
11832024-08-29T15:15:10.980ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
11842024-08-29T15:15:10.980ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11852024-08-29T15:15:10.997ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11862024-08-29T15:15:10.997ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11872024-08-29T15:15:11.016ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11882024-08-29T15:15:11.016ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
11892024-08-29T15:15:11.033ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11902024-08-29T15:15:11.033ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11912024-08-29T15:15:11.052ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11922024-08-29T15:15:11.052ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
11932024-08-29T15:15:11.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
11942024-08-29T15:15:11.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
11952024-08-29T15:15:11.085ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11962024-08-29T15:15:11.085ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
11972024-08-29T15:15:11.086ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
11982024-08-29T15:15:11.086ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
11992024-08-29T15:15:11.117ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
12002024-08-29T15:15:11.117ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
12012024-08-29T15:15:11.117ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
12022024-08-29T15:15:11.117ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
12032024-08-29T15:15:11.117ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
12042024-08-29T15:15:11.117ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 175548 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = c552dd24-a94d-4ea4-b71c-162092a39fb3 response_code = 500 uri = /omicron-zones
12052024-08-29T15:15:11.117ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c552dd24-a94d-4ea4-b71c-162092a39fb3", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:11 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c552dd24-a94d-4ea4-b71c-162092a39fb3" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 2.671100349s sled_address = [fd00:1122:3344:101::1]:12345
12062024-08-29T15:15:13.797ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12072024-08-29T15:15:13.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12082024-08-29T15:15:13.797ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12092024-08-29T15:15:13.819ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12102024-08-29T15:15:13.819ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
12112024-08-29T15:15:13.840ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12122024-08-29T15:15:13.840ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12132024-08-29T15:15:13.859ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12142024-08-29T15:15:13.859ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12152024-08-29T15:15:13.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12162024-08-29T15:15:13.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
12172024-08-29T15:15:13.896ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12182024-08-29T15:15:13.896ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12192024-08-29T15:15:13.915ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12202024-08-29T15:15:13.915ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12212024-08-29T15:15:13.933ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12222024-08-29T15:15:13.933ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12232024-08-29T15:15:13.952ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12242024-08-29T15:15:13.952ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
12252024-08-29T15:15:13.952ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
12262024-08-29T15:15:13.952ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12272024-08-29T15:15:13.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
12282024-08-29T15:15:13.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
12292024-08-29T15:15:13.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
12302024-08-29T15:15:13.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
12312024-08-29T15:15:13.984ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
12322024-08-29T15:15:13.984ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186548 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 2c394505-4505-4f58-925c-4f835ab2e97f response_code = 500 uri = /omicron-zones
12332024-08-29T15:15:13.984ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2c394505-4505-4f58-925c-4f835ab2e97f", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2c394505-4505-4f58-925c-4f835ab2e97f" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.786444778s sled_address = [fd00:1122:3344:101::1]:12345
12342024-08-29T15:15:17.782ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12352024-08-29T15:15:17.783ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12362024-08-29T15:15:17.783ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12372024-08-29T15:15:17.804ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12382024-08-29T15:15:17.804ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
12392024-08-29T15:15:17.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12402024-08-29T15:15:17.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12412024-08-29T15:15:17.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12422024-08-29T15:15:17.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12432024-08-29T15:15:17.862ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12442024-08-29T15:15:17.862ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
12452024-08-29T15:15:17.881ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12462024-08-29T15:15:17.882ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12472024-08-29T15:15:17.900ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12482024-08-29T15:15:17.900ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12492024-08-29T15:15:17.919ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12502024-08-29T15:15:17.919ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12512024-08-29T15:15:17.937ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12522024-08-29T15:15:17.937ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
12532024-08-29T15:15:17.937ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
12542024-08-29T15:15:17.938ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12552024-08-29T15:15:17.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
12562024-08-29T15:15:17.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
12572024-08-29T15:15:17.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
12582024-08-29T15:15:17.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
12592024-08-29T15:15:17.971ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
12602024-08-29T15:15:17.971ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 188447 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 260f9be6-ae36-49cb-857c-db4d767e5b97 response_code = 500 uri = /omicron-zones
12612024-08-29T15:15:17.971ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "260f9be6-ae36-49cb-857c-db4d767e5b97", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:17 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "260f9be6-ae36-49cb-857c-db4d767e5b97" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.006830684s sled_address = [fd00:1122:3344:101::1]:12345
12622024-08-29T15:15:21.990ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12632024-08-29T15:15:21.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12642024-08-29T15:15:21.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12652024-08-29T15:15:22.011ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12662024-08-29T15:15:22.011ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
12672024-08-29T15:15:22.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12682024-08-29T15:15:22.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12692024-08-29T15:15:22.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12702024-08-29T15:15:22.049ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12712024-08-29T15:15:22.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12722024-08-29T15:15:22.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
12732024-08-29T15:15:22.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12742024-08-29T15:15:22.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12752024-08-29T15:15:22.104ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12762024-08-29T15:15:22.104ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
12772024-08-29T15:15:22.122ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12782024-08-29T15:15:22.122ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
12792024-08-29T15:15:22.141ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12802024-08-29T15:15:22.141ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
12812024-08-29T15:15:22.141ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
12822024-08-29T15:15:22.141ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
12832024-08-29T15:15:22.174ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
12842024-08-29T15:15:22.174ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
12852024-08-29T15:15:22.174ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
12862024-08-29T15:15:22.174ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
12872024-08-29T15:15:22.174ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
12882024-08-29T15:15:22.174ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183227 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 9a564963-bd75-49af-bce0-8baad62bf747 response_code = 500 uri = /omicron-zones
12892024-08-29T15:15:22.174ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9a564963-bd75-49af-bce0-8baad62bf747", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:22 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9a564963-bd75-49af-bce0-8baad62bf747" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.723906088s sled_address = [fd00:1122:3344:101::1]:12345
12902024-08-29T15:15:26.437ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.313760181s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
12912024-08-29T15:15:26.913ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
12922024-08-29T15:15:26.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12932024-08-29T15:15:26.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12942024-08-29T15:15:26.934ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12952024-08-29T15:15:26.934ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
12962024-08-29T15:15:26.955ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
12972024-08-29T15:15:26.955ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12982024-08-29T15:15:26.973ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
12992024-08-29T15:15:26.973ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13002024-08-29T15:15:26.993ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13012024-08-29T15:15:26.993ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
13022024-08-29T15:15:27.012ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13032024-08-29T15:15:27.012ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13042024-08-29T15:15:27.030ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13052024-08-29T15:15:27.030ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13062024-08-29T15:15:27.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13072024-08-29T15:15:27.049ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13082024-08-29T15:15:27.067ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13092024-08-29T15:15:27.067ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
13102024-08-29T15:15:27.067ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
13112024-08-29T15:15:27.067ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13122024-08-29T15:15:27.099ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
13132024-08-29T15:15:27.099ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
13142024-08-29T15:15:27.099ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
13152024-08-29T15:15:27.099ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
13162024-08-29T15:15:27.099ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
13172024-08-29T15:15:27.099ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 185924 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 14a1ec27-5e70-4beb-8308-f762a962b641 response_code = 500 uri = /omicron-zones
13182024-08-29T15:15:27.100ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "14a1ec27-5e70-4beb-8308-f762a962b641", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:27 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "14a1ec27-5e70-4beb-8308-f762a962b641" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 3.181316361s sled_address = [fd00:1122:3344:101::1]:12345
13192024-08-29T15:15:30.290ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13202024-08-29T15:15:30.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13212024-08-29T15:15:30.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13222024-08-29T15:15:30.312ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13232024-08-29T15:15:30.312ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
13242024-08-29T15:15:30.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13252024-08-29T15:15:30.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13262024-08-29T15:15:30.349ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13272024-08-29T15:15:30.349ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13282024-08-29T15:15:30.368ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13292024-08-29T15:15:30.368ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
13302024-08-29T15:15:30.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13312024-08-29T15:15:30.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13322024-08-29T15:15:30.404ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13332024-08-29T15:15:30.404ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13342024-08-29T15:15:30.424ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13352024-08-29T15:15:30.424ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13362024-08-29T15:15:30.442ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13372024-08-29T15:15:30.442ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
13382024-08-29T15:15:30.442ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
13392024-08-29T15:15:30.442ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13402024-08-29T15:15:30.474ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
13412024-08-29T15:15:30.474ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
13422024-08-29T15:15:30.474ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
13432024-08-29T15:15:30.474ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
13442024-08-29T15:15:30.474ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
13452024-08-29T15:15:30.474ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 183197 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 79408470-96f8-42ab-a15f-48f91fbb0cb2 response_code = 500 uri = /omicron-zones
13462024-08-29T15:15:30.474ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "79408470-96f8-42ab-a15f-48f91fbb0cb2", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:30 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "79408470-96f8-42ab-a15f-48f91fbb0cb2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 4.032582996s sled_address = [fd00:1122:3344:101::1]:12345
13472024-08-29T15:15:34.519ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13482024-08-29T15:15:34.519ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13492024-08-29T15:15:34.519ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13502024-08-29T15:15:34.540ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13512024-08-29T15:15:34.540ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
13522024-08-29T15:15:34.558ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13532024-08-29T15:15:34.559ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13542024-08-29T15:15:34.576ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13552024-08-29T15:15:34.576ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13562024-08-29T15:15:34.595ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13572024-08-29T15:15:34.595ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
13582024-08-29T15:15:34.614ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13592024-08-29T15:15:34.614ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13602024-08-29T15:15:34.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13612024-08-29T15:15:34.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13622024-08-29T15:15:34.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13632024-08-29T15:15:34.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13642024-08-29T15:15:34.668ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13652024-08-29T15:15:34.668ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
13662024-08-29T15:15:34.668ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
13672024-08-29T15:15:34.668ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13682024-08-29T15:15:34.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
13692024-08-29T15:15:34.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
13702024-08-29T15:15:34.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
13712024-08-29T15:15:34.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
13722024-08-29T15:15:34.700ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
13732024-08-29T15:15:34.700ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181281 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = ac16d1da-b833-4958-bc98-63cc320dbb62 response_code = 500 uri = /omicron-zones
13742024-08-29T15:15:34.701ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ac16d1da-b833-4958-bc98-63cc320dbb62", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ac16d1da-b833-4958-bc98-63cc320dbb62" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 10.497327632s sled_address = [fd00:1122:3344:101::1]:12345
13752024-08-29T15:15:35.118ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:50021
13762024-08-29T15:15:35.118ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:50021 req_id = a857a108-0c5a-45b1-b513-d9dd99a18f47 response_code = 200 uri = /baseboard
13772024-08-29T15:15:42.832ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 2.468100608s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
13782024-08-29T15:15:45.228ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
13792024-08-29T15:15:45.228ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13802024-08-29T15:15:45.228ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13812024-08-29T15:15:45.250ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13822024-08-29T15:15:45.250ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
13832024-08-29T15:15:45.270ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
13842024-08-29T15:15:45.270ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13852024-08-29T15:15:45.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13862024-08-29T15:15:45.289ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13872024-08-29T15:15:45.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13882024-08-29T15:15:45.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
13892024-08-29T15:15:45.326ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13902024-08-29T15:15:45.326ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13912024-08-29T15:15:45.345ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13922024-08-29T15:15:45.345ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
13932024-08-29T15:15:45.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
13942024-08-29T15:15:45.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
13952024-08-29T15:15:45.381ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13962024-08-29T15:15:45.381ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
13972024-08-29T15:15:45.382ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
13982024-08-29T15:15:45.382ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
13992024-08-29T15:15:45.415ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
14002024-08-29T15:15:45.415ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
14012024-08-29T15:15:45.415ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
14022024-08-29T15:15:45.415ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
14032024-08-29T15:15:45.415ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
14042024-08-29T15:15:45.415ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 186306 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 01432e4b-f743-4c51-82a5-e9a183490fc6 response_code = 500 uri = /omicron-zones
14052024-08-29T15:15:45.415ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "01432e4b-f743-4c51-82a5-e9a183490fc6", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "01432e4b-f743-4c51-82a5-e9a183490fc6" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 12.613846732s sled_address = [fd00:1122:3344:101::1]:12345
14062024-08-29T15:15:58.064ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14072024-08-29T15:15:58.065ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14082024-08-29T15:15:58.065ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14092024-08-29T15:15:58.086ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14102024-08-29T15:15:58.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
14112024-08-29T15:15:58.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14122024-08-29T15:15:58.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14132024-08-29T15:15:58.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14142024-08-29T15:15:58.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14152024-08-29T15:15:58.142ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14162024-08-29T15:15:58.142ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
14172024-08-29T15:15:58.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14182024-08-29T15:15:58.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14192024-08-29T15:15:58.178ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14202024-08-29T15:15:58.178ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14212024-08-29T15:15:58.195ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14222024-08-29T15:15:58.195ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
14232024-08-29T15:15:58.213ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14242024-08-29T15:15:58.213ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
14252024-08-29T15:15:58.213ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
14262024-08-29T15:15:58.213ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14272024-08-29T15:15:58.243ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
14282024-08-29T15:15:58.243ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
14292024-08-29T15:15:58.243ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
14302024-08-29T15:15:58.243ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
14312024-08-29T15:15:58.243ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
14322024-08-29T15:15:58.243ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 178026 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 04dd8666-f5da-4a79-8972-780556451c22 response_code = 500 uri = /omicron-zones
14332024-08-29T15:15:58.243ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "04dd8666-f5da-4a79-8972-780556451c22", "content-length": "124", "date": "Thu, 29 Aug 2024 15:15:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "04dd8666-f5da-4a79-8972-780556451c22" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 15.299847273s sled_address = [fd00:1122:3344:101::1]:12345
14342024-08-29T15:16:00.402ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.607903928s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
14352024-08-29T15:16:05.235ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:47214
14362024-08-29T15:16:05.235ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 57 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:47214 req_id = 98a2376a-aaef-414a-bdb2-0c37482884d8 response_code = 200 uri = /baseboard
14372024-08-29T15:16:13.585ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14382024-08-29T15:16:13.586ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14392024-08-29T15:16:13.586ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14402024-08-29T15:16:13.607ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14412024-08-29T15:16:13.607ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
14422024-08-29T15:16:13.626ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14432024-08-29T15:16:13.626ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14442024-08-29T15:16:13.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14452024-08-29T15:16:13.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14462024-08-29T15:16:13.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14472024-08-29T15:16:13.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
14482024-08-29T15:16:13.681ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14492024-08-29T15:16:13.681ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14502024-08-29T15:16:13.699ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14512024-08-29T15:16:13.700ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14522024-08-29T15:16:13.717ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14532024-08-29T15:16:13.717ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
14542024-08-29T15:16:13.736ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14552024-08-29T15:16:13.736ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
14562024-08-29T15:16:13.736ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
14572024-08-29T15:16:13.736ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14582024-08-29T15:16:13.770ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
14592024-08-29T15:16:13.770ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
14602024-08-29T15:16:13.770ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
14612024-08-29T15:16:13.770ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
14622024-08-29T15:16:13.770ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
14632024-08-29T15:16:13.770ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 184378 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = dd2a1f2f-f49e-4223-80cd-b2b4b73496ff response_code = 500 uri = /omicron-zones
14642024-08-29T15:16:13.770ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "dd2a1f2f-f49e-4223-80cd-b2b4b73496ff", "content-length": "124", "date": "Thu, 29 Aug 2024 15:16:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "dd2a1f2f-f49e-4223-80cd-b2b4b73496ff" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 7.230124239s sled_address = [fd00:1122:3344:101::1]:12345
14652024-08-29T15:16:20.100ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 10.243453253s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
14662024-08-29T15:16:21.022ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14672024-08-29T15:16:21.022ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14682024-08-29T15:16:21.022ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14692024-08-29T15:16:21.044ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14702024-08-29T15:16:21.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
14712024-08-29T15:16:21.063ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14722024-08-29T15:16:21.063ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14732024-08-29T15:16:21.082ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14742024-08-29T15:16:21.082ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14752024-08-29T15:16:21.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14762024-08-29T15:16:21.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
14772024-08-29T15:16:21.118ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14782024-08-29T15:16:21.118ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14792024-08-29T15:16:21.136ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14802024-08-29T15:16:21.136ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
14812024-08-29T15:16:21.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
14822024-08-29T15:16:21.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
14832024-08-29T15:16:21.172ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14842024-08-29T15:16:21.172ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
14852024-08-29T15:16:21.172ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
14862024-08-29T15:16:21.172ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
14872024-08-29T15:16:21.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
14882024-08-29T15:16:21.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
14892024-08-29T15:16:21.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
14902024-08-29T15:16:21.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
14912024-08-29T15:16:21.204ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
14922024-08-29T15:16:21.204ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181996 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = fd55732d-1523-431d-a6ef-8a81c8c427a2 response_code = 500 uri = /omicron-zones
14932024-08-29T15:16:21.205ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fd55732d-1523-431d-a6ef-8a81c8c427a2", "content-length": "124", "date": "Thu, 29 Aug 2024 15:16:21 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fd55732d-1523-431d-a6ef-8a81c8c427a2" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 13.356932672s sled_address = [fd00:1122:3344:101::1]:12345
14942024-08-29T15:16:34.599ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
14952024-08-29T15:16:34.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14962024-08-29T15:16:34.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14972024-08-29T15:16:34.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
14982024-08-29T15:16:34.621ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
14992024-08-29T15:16:34.640ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15002024-08-29T15:16:34.640ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15012024-08-29T15:16:34.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15022024-08-29T15:16:34.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15032024-08-29T15:16:34.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15042024-08-29T15:16:34.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
15052024-08-29T15:16:34.695ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15062024-08-29T15:16:34.695ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15072024-08-29T15:16:34.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15082024-08-29T15:16:34.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15092024-08-29T15:16:34.731ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15102024-08-29T15:16:34.731ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
15112024-08-29T15:16:34.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15122024-08-29T15:16:34.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
15132024-08-29T15:16:34.749ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
15142024-08-29T15:16:34.749ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15152024-08-29T15:16:34.781ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
15162024-08-29T15:16:34.781ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
15172024-08-29T15:16:34.781ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
15182024-08-29T15:16:34.781ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
15192024-08-29T15:16:34.781ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
15202024-08-29T15:16:34.781ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 182066 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = b1227758-043f-43e5-be5a-46dd999b5cd7 response_code = 500 uri = /omicron-zones
15212024-08-29T15:16:34.781ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b1227758-043f-43e5-be5a-46dd999b5cd7", "content-length": "124", "date": "Thu, 29 Aug 2024 15:16:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b1227758-043f-43e5-be5a-46dd999b5cd7" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 17.750605211s sled_address = [fd00:1122:3344:101::1]:12345
15222024-08-29T15:16:35.354ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:49216
15232024-08-29T15:16:35.354ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:49216 req_id = 8656a3b5-4c05-4166-870a-8f7aa7c7f2b5 response_code = 200 uri = /baseboard
15242024-08-29T15:16:45.450ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 21.684470562s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
15252024-08-29T15:16:52.582ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
15262024-08-29T15:16:52.582ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15272024-08-29T15:16:52.582ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15282024-08-29T15:16:52.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15292024-08-29T15:16:52.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
15302024-08-29T15:16:52.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15312024-08-29T15:16:52.623ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15322024-08-29T15:16:52.640ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15332024-08-29T15:16:52.640ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15342024-08-29T15:16:52.658ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15352024-08-29T15:16:52.658ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
15362024-08-29T15:16:52.677ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15372024-08-29T15:16:52.677ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15382024-08-29T15:16:52.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15392024-08-29T15:16:52.696ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15402024-08-29T15:16:52.714ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15412024-08-29T15:16:52.714ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
15422024-08-29T15:16:52.731ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15432024-08-29T15:16:52.731ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
15442024-08-29T15:16:52.732ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
15452024-08-29T15:16:52.732ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15462024-08-29T15:16:52.764ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
15472024-08-29T15:16:52.764ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
15482024-08-29T15:16:52.764ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
15492024-08-29T15:16:52.764ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
15502024-08-29T15:16:52.764ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:3254 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
15512024-08-29T15:16:52.764ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e", TimeNotSynchronized), ("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316", TimeNotSynchronized), ("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33", TimeNotSynchronized), ("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875", TimeNotSynchronized), ("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:902 latency_us = 181341 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 4862bad0-989d-4b77-acfa-d1709eb43602 response_code = 500 uri = /omicron-zones
15522024-08-29T15:16:52.764ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4862bad0-989d-4b77-acfa-d1709eb43602", "content-length": "124", "date": "Thu, 29 Aug 2024 15:16:52 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4862bad0-989d-4b77-acfa-d1709eb43602" } file = sled-agent/src/rack_setup/service.rs:475 retry_after = 32.567222173s sled_address = [fd00:1122:3344:101::1]:12345
15532024-08-29T15:17:05.469ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:42234
15542024-08-29T15:17:05.469ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:42234 req_id = 24b29b66-5472-4159-ba7d-ed23287d0f46 response_code = 200 uri = /baseboard
15552024-08-29T15:17:22.269ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 24.73035272s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
15562024-08-29T15:17:25.421ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
15572024-08-29T15:17:25.421ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15582024-08-29T15:17:25.421ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15592024-08-29T15:17:25.444ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15602024-08-29T15:17:25.444ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
15612024-08-29T15:17:25.462ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
15622024-08-29T15:17:25.463ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15632024-08-29T15:17:25.482ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15642024-08-29T15:17:25.482ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15652024-08-29T15:17:25.502ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15662024-08-29T15:17:25.503ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
15672024-08-29T15:17:25.521ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15682024-08-29T15:17:25.521ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15692024-08-29T15:17:25.541ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15702024-08-29T15:17:25.541ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
15712024-08-29T15:17:25.561ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
15722024-08-29T15:17:25.561ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
15732024-08-29T15:17:25.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15742024-08-29T15:17:25.579ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
15752024-08-29T15:17:25.579ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
15762024-08-29T15:17:25.579ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15772024-08-29T15:17:25.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15782024-08-29T15:17:25.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15792024-08-29T15:17:25.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15802024-08-29T15:17:25.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15812024-08-29T15:17:25.661ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
15822024-08-29T15:17:25.752ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e file = illumos-utils/src/zone.rs:316
15832024-08-29T15:17:25.756ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875 file = illumos-utils/src/zone.rs:316
15842024-08-29T15:17:25.760ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de file = illumos-utils/src/zone.rs:316
15852024-08-29T15:17:25.764ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316 file = illumos-utils/src/zone.rs:316
15862024-08-29T15:17:25.769ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33 file = illumos-utils/src/zone.rs:316
15872024-08-29T15:17:25.773ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e file = illumos-utils/src/zone.rs:354
15882024-08-29T15:17:25.778ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875 file = illumos-utils/src/zone.rs:354
15892024-08-29T15:17:25.782ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de file = illumos-utils/src/zone.rs:354
15902024-08-29T15:17:25.787ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316 file = illumos-utils/src/zone.rs:354
15912024-08-29T15:17:25.792ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33 file = illumos-utils/src/zone.rs:354
15922024-08-29T15:17:28.652ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService8'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='5dc3d2e8-7d3d-4338-9928-132828621316'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::7]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::7]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15932024-08-29T15:17:28.652ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService5'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='0dac43bc-7b48-477d-9f9f-6fd3fe599f5e'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::5]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::5]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15942024-08-29T15:17:28.652ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
15952024-08-29T15:17:28.658ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
15962024-08-29T15:17:28.716ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService6'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::4'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='70a742d6-87c1-4c50-8096-5f2e5549f875'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::4]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::4]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15972024-08-29T15:17:28.716ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
15982024-08-29T15:17:28.725ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService9'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::3'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='6d2f290a-abdf-4ede-9ef2-830ec6ce9b33'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::3]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::3]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
15992024-08-29T15:17:28.725ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16002024-08-29T15:17:28.745ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService7'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value='6a3e1632-3f67-4c67-a09b-2ee7c055f6de'/> <propval type="astring" name="cockroach_address" value='[fd00:1122:3344:101::6]:32221'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::6]:32222'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
16012024-08-29T15:17:28.745ZINFOSledAgent (ServiceManager): Booting oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16022024-08-29T15:17:28.810ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 43.129385ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16032024-08-29T15:17:28.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 26.573782ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16042024-08-29T15:17:28.853ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 43.925953ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16052024-08-29T15:17:28.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 50.507784ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16062024-08-29T15:17:28.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 72.130124ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16072024-08-29T15:17:28.963ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 62.297876ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16082024-08-29T15:17:29.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 194.634023ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16092024-08-29T15:17:29.903ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 79.586642ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16102024-08-29T15:17:31.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 74.841388ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16112024-08-29T15:17:31.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 217.10954ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16122024-08-29T15:17:31.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 77.9811ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16132024-08-29T15:17:31.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 424.351178ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16142024-08-29T15:17:31.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 184.546234ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16152024-08-29T15:17:31.791ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 91.350338ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16162024-08-29T15:17:31.842ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 264.453475ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16172024-08-29T15:17:31.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 165.167171ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16182024-08-29T15:17:31.907ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 286.189968ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16192024-08-29T15:17:31.977ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 280.247183ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16202024-08-29T15:17:32.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 276.995414ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16212024-08-29T15:17:32.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 573.428016ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16222024-08-29T15:17:32.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 753.607736ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16232024-08-29T15:17:32.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 1.169873683s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16242024-08-29T15:17:32.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 475.498528ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16252024-08-29T15:17:32.363ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 483.869852ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16262024-08-29T15:17:32.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 901.582681ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16272024-08-29T15:17:32.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 555.949166ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16282024-08-29T15:17:32.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 1.240512775s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16292024-08-29T15:17:32.964ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 760.317013ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16302024-08-29T15:17:33.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.430098043s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16312024-08-29T15:17:33.390ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 867.496446ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16322024-08-29T15:17:33.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 543.840089ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16332024-08-29T15:17:33.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 862.718664ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16342024-08-29T15:17:34.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 1.472606006s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16352024-08-29T15:17:34.167ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 590.807355ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16362024-08-29T15:17:34.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 883.259699ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16372024-08-29T15:17:34.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 1.305317161s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16382024-08-29T15:17:34.758ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.01262188s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16392024-08-29T15:17:34.768ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 1.384612632s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16402024-08-29T15:17:35.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 1.302232228s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16412024-08-29T15:17:35.556ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53764
16422024-08-29T15:17:35.556ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53764 req_id = 731b0e93-c93c-4013-a851-c1599d30a4d8 response_code = 200 uri = /baseboard
16432024-08-29T15:17:35.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 511.401939ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16442024-08-29T15:17:35.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 949.724495ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16452024-08-29T15:17:35.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 1.432813346s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16462024-08-29T15:17:36.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 704.205216ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16472024-08-29T15:17:36.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 726.699692ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16482024-08-29T15:17:36.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 600.724781ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16492024-08-29T15:17:36.739ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 527.325812ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16502024-08-29T15:17:36.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 1.067150647s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16512024-08-29T15:17:36.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 568.854797ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16522024-08-29T15:17:37.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316") failed: Property not found. retry in 1.319081848s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16532024-08-29T15:17:37.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.361893154s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16542024-08-29T15:17:37.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e") failed: Property not found. retry in 757.506357ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16552024-08-29T15:17:37.476ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33") failed: Property not found. retry in 1.412273509s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16562024-08-29T15:17:37.902ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 1.099652738s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16572024-08-29T15:17:38.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 937.213888ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16582024-08-29T15:17:39.028ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 927.275596ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16592024-08-29T15:17:39.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.255926741s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16602024-08-29T15:17:39.966ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 694.244503ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16612024-08-29T15:17:41.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 880.992795ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16622024-08-29T15:17:41.158ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 1.1871751s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16632024-08-29T15:17:42.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 507.172547ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16642024-08-29T15:17:42.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 983.160715ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16652024-08-29T15:17:42.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 955.337369ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16662024-08-29T15:17:43.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 551.332329ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16672024-08-29T15:17:43.513ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 845.328482ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16682024-08-29T15:17:43.908ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 681.396349ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16692024-08-29T15:17:44.369ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.24736394s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16702024-08-29T15:17:44.598ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 665.676033ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16712024-08-29T15:17:45.273ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 937.791085ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16722024-08-29T15:17:45.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.422654582s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16732024-08-29T15:17:46.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 1.116622914s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16742024-08-29T15:17:47.058ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875") failed: Property not found. retry in 1.370573841s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16752024-08-29T15:17:47.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 577.721189ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16762024-08-29T15:17:47.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de") failed: Property not found. retry in 531.135141ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16772024-08-29T15:17:48.485ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_0dac43bc-7b48-477d-9f9f-6fd3fe599f5e
16782024-08-29T15:17:48.485ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_70a742d6-87c1-4c50-8096-5f2e5549f875
16792024-08-29T15:17:48.485ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_6a3e1632-3f67-4c67-a09b-2ee7c055f6de
16802024-08-29T15:17:48.485ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_5dc3d2e8-7d3d-4338-9928-132828621316
16812024-08-29T15:17:48.485ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_cockroachdb_6d2f290a-abdf-4ede-9ef2-830ec6ce9b33
16822024-08-29T15:17:48.485ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16832024-08-29T15:17:48.486ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
16842024-08-29T15:17:48.487ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 23065599 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:41420 req_id = 393d84ba-3987-42a1-9baa-ad6590896c95 response_code = 204 uri = /omicron-zones
16852024-08-29T15:17:48.520ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65093
16862024-08-29T15:17:48.549ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::5]:32221 file = sled-agent/src/services.rs:3686
16872024-08-29T15:17:49.393ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3704
16882024-08-29T15:18:12.437ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3727
16892024-08-29T15:18:12.437ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 23917746 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:65093 req_id = 17f05b7b-48d8-4b00-bdab-314d95df55bc response_code = 204 uri = /cockroachdb
16902024-08-29T15:18:12.438ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44615
16912024-08-29T15:18:12.438ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44615 req_id = a7fffcd1-51dc-4bc7-a1b1-955a2df5477f response_code = 200 uri = /baseboard
16922024-08-29T15:18:12.443ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 89.726462634s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:424 sled_id = b9c40332-a231-42db-adf0-b267d663a311
16932024-08-29T15:18:12.473ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: 285489bf-dd9f-4685-a3f1-46733c794cfb, underlay_address: fd00:1122:3344:1::1, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, underlay_address: fd00:1122:3344:2::1, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: c7342971-675e-4b24-995a-35efbafcac87, underlay_address: fd00:1122:3344:3::1, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25, underlay_address: fd00:1122:3344:101::1b, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: b8c196d9-a607-40a2-9036-1e2d2f60dc07, kind: Service { id: 43a6fa50-4493-4748-921d-7ff7c23c0e25 }, name: Name("ntp-43a6fa50-4493-4748-921d-7ff7c23c0e25"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 223, 79])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, underlay_address: fd00:1122:3344:101::3, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 70a742d6-87c1-4c50-8096-5f2e5549f875, underlay_address: fd00:1122:3344:101::4, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, underlay_address: fd00:1122:3344:101::5, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, underlay_address: fd00:1122:3344:101::6, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 5dc3d2e8-7d3d-4338-9928-132828621316, underlay_address: fd00:1122:3344:101::7, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }, OmicronZoneConfig { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974, underlay_address: fd00:1122:3344:101::8, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9, underlay_address: fd00:1122:3344:101::9, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] } } }, OmicronZoneConfig { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9, underlay_address: fd00:1122:3344:101::a, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5, underlay_address: fd00:1122:3344:101::b, filesystem_pool: Some(ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: a752201f-8e19-45ac-9901-65317e43fc2e, underlay_address: fd00:1122:3344:101::c, filesystem_pool: Some(ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External }), zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 776ef919-ac0a-4b14-9dfe-deac75d16696, underlay_address: fd00:1122:3344:101::d, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 709ff80f-750f-45c6-b668-97969f356dc0, underlay_address: fd00:1122:3344:101::e, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 4c2f62ac-cb40-467c-ba9f-de7aa05f3691, underlay_address: fd00:1122:3344:101::f, filesystem_pool: Some(ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: 45d4b7eb-c738-4794-9125-01d0ae07feb2, underlay_address: fd00:1122:3344:101::10, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: bbf9c6ad-fa23-470f-b608-67105cf521a9, underlay_address: fd00:1122:3344:101::11, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 5b880f4d-eaa0-4b77-919b-817d6a980247, underlay_address: fd00:1122:3344:101::12, filesystem_pool: Some(ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External } } } }, OmicronZoneConfig { id: aa3cf0ac-1c68-4e1c-a127-490862a19932, underlay_address: fd00:1122:3344:101::13, filesystem_pool: Some(ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External } } } }, OmicronZoneConfig { id: 5a83427f-50d0-47ad-8765-fad7f4c19873, underlay_address: fd00:1122:3344:101::14, filesystem_pool: Some(ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External } } } }, OmicronZoneConfig { id: f389808b-a883-46ea-a396-af6431b5b6e3, underlay_address: fd00:1122:3344:101::15, filesystem_pool: Some(ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External } } } }, OmicronZoneConfig { id: 23eb92fe-33fd-4e95-a819-b3349525e67a, underlay_address: fd00:1122:3344:101::16, filesystem_pool: Some(ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External } } } }, OmicronZoneConfig { id: d98c3f2a-4c9c-4655-9407-3839f53cf851, underlay_address: fd00:1122:3344:101::17, filesystem_pool: Some(ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External } } } }, OmicronZoneConfig { id: 57f24d92-52ba-4db7-a1b1-c2a25e015a03, underlay_address: fd00:1122:3344:101::18, filesystem_pool: Some(ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9becf607-a424-4af6-bd2e-178c196b8348, underlay_address: fd00:1122:3344:101::19, filesystem_pool: Some(ZpoolName { id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool), kind: External } } } }, OmicronZoneConfig { id: 3da8a36e-1727-4d21-9b85-8be498e95b8c, underlay_address: fd00:1122:3344:101::1a, filesystem_pool: Some(ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External }), zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:438 sled_address = [fd00:1122:3344:101::1]:12345
16942024-08-29T15:18:12.473ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:65026
16952024-08-29T15:18:12.474ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16962024-08-29T15:18:12.474ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 285489bf-dd9f-4685-a3f1-46733c794cfb, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16972024-08-29T15:18:12.498ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
16982024-08-29T15:18:12.498ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: ccbc07cd-fbf0-43da-b57b-17b3debd7838, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: InternalDns }, responder: ... }
16992024-08-29T15:18:12.520ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17002024-08-29T15:18:12.520ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: c7342971-675e-4b24-995a-35efbafcac87, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17012024-08-29T15:18:12.542ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17022024-08-29T15:18:12.542ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6d2f290a-abdf-4ede-9ef2-830ec6ce9b33, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17032024-08-29T15:18:12.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17042024-08-29T15:18:12.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 70a742d6-87c1-4c50-8096-5f2e5549f875, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Cockroach }, responder: ... }
17052024-08-29T15:18:12.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17062024-08-29T15:18:12.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 0dac43bc-7b48-477d-9f9f-6fd3fe599f5e, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17072024-08-29T15:18:12.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17082024-08-29T15:18:12.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 6a3e1632-3f67-4c67-a09b-2ee7c055f6de, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Cockroach }, responder: ... }
17092024-08-29T15:18:12.619ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }) file = sled-storage/src/manager.rs:451
17102024-08-29T15:18:12.619ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5dc3d2e8-7d3d-4338-9928-132828621316, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Cockroach }, responder: ... }
17112024-08-29T15:18:12.638ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 45f571be-936e-48ca-b02e-6bbc2b2f1974, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17122024-08-29T15:18:12.638ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 45f571be-936e-48ca-b02e-6bbc2b2f1974, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17132024-08-29T15:18:12.773ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:451
17142024-08-29T15:18:12.773ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17152024-08-29T15:18:12.970ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 709ff80f-750f-45c6-b668-97969f356dc0, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:451
17162024-08-29T15:18:12.970ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 709ff80f-750f-45c6-b668-97969f356dc0, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17172024-08-29T15:18:13.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5b880f4d-eaa0-4b77-919b-817d6a980247, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17182024-08-29T15:18:13.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5b880f4d-eaa0-4b77-919b-817d6a980247, dataset_name: DatasetName { pool_name: ZpoolName { id: 05effe78-0c08-40b4-916b-cde344744ec3 (zpool), kind: External }, kind: Crucible }, responder: ... }
17192024-08-29T15:18:13.186ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: aa3cf0ac-1c68-4e1c-a127-490862a19932, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17202024-08-29T15:18:13.186ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: aa3cf0ac-1c68-4e1c-a127-490862a19932, dataset_name: DatasetName { pool_name: ZpoolName { id: 866d818f-82bb-444c-992a-1351dec91efa (zpool), kind: External }, kind: Crucible }, responder: ... }
17212024-08-29T15:18:13.320ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 5a83427f-50d0-47ad-8765-fad7f4c19873, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17222024-08-29T15:18:13.320ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 5a83427f-50d0-47ad-8765-fad7f4c19873, dataset_name: DatasetName { pool_name: ZpoolName { id: 4fbbfff0-e175-4404-8cd9-acfc358b7140 (zpool), kind: External }, kind: Crucible }, responder: ... }
17232024-08-29T15:18:13.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: f389808b-a883-46ea-a396-af6431b5b6e3, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17242024-08-29T15:18:13.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: f389808b-a883-46ea-a396-af6431b5b6e3, dataset_name: DatasetName { pool_name: ZpoolName { id: 6eaf0007-0aa9-44b6-8bac-b01569f4c6a3 (zpool), kind: External }, kind: Crucible }, responder: ... }
17252024-08-29T15:18:13.566ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 23eb92fe-33fd-4e95-a819-b3349525e67a, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17262024-08-29T15:18:13.566ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 23eb92fe-33fd-4e95-a819-b3349525e67a, dataset_name: DatasetName { pool_name: ZpoolName { id: 775439b8-d6ea-4b72-9708-a30c4c7c6d6f (zpool), kind: External }, kind: Crucible }, responder: ... }
17272024-08-29T15:18:13.684ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d98c3f2a-4c9c-4655-9407-3839f53cf851, dataset_name: DatasetName { pool_name: ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17282024-08-29T15:18:13.684ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: d98c3f2a-4c9c-4655-9407-3839f53cf851, dataset_name: DatasetName { pool_name: ZpoolName { id: e4f77dae-534d-4b73-b38a-890d525d3747 (zpool), kind: External }, kind: Crucible }, responder: ... }
17292024-08-29T15:18:13.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 57f24d92-52ba-4db7-a1b1-c2a25e015a03, dataset_name: DatasetName { pool_name: ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17302024-08-29T15:18:13.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 57f24d92-52ba-4db7-a1b1-c2a25e015a03, dataset_name: DatasetName { pool_name: ZpoolName { id: 1b55399c-9dbb-4d32-bd51-426a7996ea24 (zpool), kind: External }, kind: Crucible }, responder: ... }
17312024-08-29T15:18:13.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9becf607-a424-4af6-bd2e-178c196b8348, dataset_name: DatasetName { pool_name: ZpoolName { id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17322024-08-29T15:18:13.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 9becf607-a424-4af6-bd2e-178c196b8348, dataset_name: DatasetName { pool_name: ZpoolName { id: aad99d77-5c0c-4ae7-8cc1-ced403d6b98a (zpool), kind: External }, kind: Crucible }, responder: ... }
17332024-08-29T15:18:14.150ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 3da8a36e-1727-4d21-9b85-8be498e95b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:451
17342024-08-29T15:18:14.150ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:1065 request = NewFilesystemRequest { dataset_id: 3da8a36e-1727-4d21-9b85-8be498e95b8c, dataset_name: DatasetName { pool_name: ZpoolName { id: f4912882-4c2a-4c0d-b986-5cbff960587e (zpool), kind: External }, kind: Crucible }, responder: ... }
17352024-08-29T15:18:14.282ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17362024-08-29T15:18:14.282ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
17372024-08-29T15:18:14.282ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
17382024-08-29T15:18:14.283ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17392024-08-29T15:18:14.404ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
17402024-08-29T15:18:14.404ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17412024-08-29T15:18:14.416ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
17422024-08-29T15:18:14.418ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
17432024-08-29T15:18:14.490ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17442024-08-29T15:18:14.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17452024-08-29T15:18:14.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17462024-08-29T15:18:14.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17472024-08-29T15:18:14.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17482024-08-29T15:18:14.492ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17492024-08-29T15:18:14.639ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
17502024-08-29T15:18:14.708ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:260
17512024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17522024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17532024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17542024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17552024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17562024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17572024-08-29T15:18:14.708ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
17582024-08-29T15:18:14.708ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696 file = illumos-utils/src/zone.rs:316
17592024-08-29T15:18:14.714ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0 file = illumos-utils/src/zone.rs:316
17602024-08-29T15:18:14.719ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247 file = illumos-utils/src/zone.rs:316
17612024-08-29T15:18:14.723ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873 file = illumos-utils/src/zone.rs:316
17622024-08-29T15:18:14.728ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851 file = illumos-utils/src/zone.rs:316
17632024-08-29T15:18:14.733ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3 file = illumos-utils/src/zone.rs:316
17642024-08-29T15:18:14.739ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a file = illumos-utils/src/zone.rs:316
17652024-08-29T15:18:14.883ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696 file = illumos-utils/src/zone.rs:354
17662024-08-29T15:18:14.890ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0 file = illumos-utils/src/zone.rs:354
17672024-08-29T15:18:14.895ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247 file = illumos-utils/src/zone.rs:354
17682024-08-29T15:18:14.901ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873 file = illumos-utils/src/zone.rs:354
17692024-08-29T15:18:14.906ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851 file = illumos-utils/src/zone.rs:354
17702024-08-29T15:18:14.912ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3 file = illumos-utils/src/zone.rs:354
17712024-08-29T15:18:14.916ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a file = illumos-utils/src/zone.rs:354
17722024-08-29T15:18:14.921ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2 file = illumos-utils/src/zone.rs:316
17732024-08-29T15:18:14.927ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9 file = illumos-utils/src/zone.rs:316
17742024-08-29T15:18:14.932ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691 file = illumos-utils/src/zone.rs:316
17752024-08-29T15:18:14.937ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932 file = illumos-utils/src/zone.rs:316
17762024-08-29T15:18:14.943ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348 file = illumos-utils/src/zone.rs:316
17772024-08-29T15:18:14.948ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c file = illumos-utils/src/zone.rs:316
17782024-08-29T15:18:14.953ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03 file = illumos-utils/src/zone.rs:316
17792024-08-29T15:18:14.996ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
17802024-08-29T15:18:14.997ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2 file = illumos-utils/src/zone.rs:354
17812024-08-29T15:18:15.002ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9 file = illumos-utils/src/zone.rs:354
17822024-08-29T15:18:15.007ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691 file = illumos-utils/src/zone.rs:354
17832024-08-29T15:18:15.012ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932 file = illumos-utils/src/zone.rs:354
17842024-08-29T15:18:15.017ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348 file = illumos-utils/src/zone.rs:354
17852024-08-29T15:18:15.022ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c file = illumos-utils/src/zone.rs:354
17862024-08-29T15:18:15.027ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03 file = illumos-utils/src/zone.rs:354
17872024-08-29T15:18:15.068ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
17882024-08-29T15:18:15.102ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
17892024-08-29T15:18:15.136ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
17902024-08-29T15:18:15.139ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
17912024-08-29T15:18:15.139ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
17922024-08-29T15:18:15.173ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte1", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 134, 65]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17932024-08-29T15:18:15.173ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
17942024-08-29T15:18:15.219ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:277
17952024-08-29T15:18:15.221ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
17962024-08-29T15:18:15.221ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
17972024-08-29T15:18:15.272ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 132, 224]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
17982024-08-29T15:18:15.272ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
17992024-08-29T15:18:15.274ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18002024-08-29T15:18:15.274ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18012024-08-29T15:18:15.328ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte3", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 235, 245]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 } }) }
18022024-08-29T15:18:15.328ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = nexus
18032024-08-29T15:18:15.332ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18042024-08-29T15:18:15.332ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18052024-08-29T15:18:15.440ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte4", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 217, 255]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
18062024-08-29T15:18:15.440ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
18072024-08-29T15:18:15.440ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:83
18082024-08-29T15:18:15.440ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:90
18092024-08-29T15:18:15.441ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:302
18102024-08-29T15:18:15.441ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:194
18112024-08-29T15:18:15.493ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte5", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 188, 52]), slot: 0, vni: Vni { inner: 100 }, subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 } }) }
18122024-08-29T15:18:15.493ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1234 zone_type = external_dns
18132024-08-29T15:18:15.494ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.166 file = clients/dpd-client/src/lib.rs:118
18142024-08-29T15:18:15.495ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18152024-08-29T15:18:15.541ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:83
18162024-08-29T15:18:15.541ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:90
18172024-08-29T15:18:15.541ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:83
18182024-08-29T15:18:15.541ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:90
18192024-08-29T15:18:15.541ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:83
18202024-08-29T15:18:15.541ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:90
18212024-08-29T15:18:15.542ZINFOSledAgent (ServiceManager): no nat entry found for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:83
18222024-08-29T15:18:15.542ZINFOSledAgent (ServiceManager): creating nat entry for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:90
18232024-08-29T15:18:15.542ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.167 file = clients/dpd-client/src/lib.rs:118
18242024-08-29T15:18:15.545ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.168 file = clients/dpd-client/src/lib.rs:118
18252024-08-29T15:18:15.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18262024-08-29T15:18:15.548ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.164 file = clients/dpd-client/src/lib.rs:118
18272024-08-29T15:18:15.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18282024-08-29T15:18:15.552ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18292024-08-29T15:18:15.552ZINFOSledAgent (ServiceManager): creation of nat entry successful for: 10.151.2.165 file = clients/dpd-client/src/lib.rs:118
18302024-08-29T15:18:15.733ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
18312024-08-29T15:18:15.734ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9 file = illumos-utils/src/zone.rs:316
18322024-08-29T15:18:15.810ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5 file = illumos-utils/src/zone.rs:316
18332024-08-29T15:18:15.831ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e file = illumos-utils/src/zone.rs:316
18342024-08-29T15:18:15.855ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974 file = illumos-utils/src/zone.rs:316
18352024-08-29T15:18:15.876ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9 file = illumos-utils/src/zone.rs:354
18362024-08-29T15:18:15.893ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9 file = illumos-utils/src/zone.rs:316
18372024-08-29T15:18:15.907ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5 file = illumos-utils/src/zone.rs:354
18382024-08-29T15:18:15.922ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e file = illumos-utils/src/zone.rs:354
18392024-08-29T15:18:15.937ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974 file = illumos-utils/src/zone.rs:354
18402024-08-29T15:18:15.954ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9 file = illumos-utils/src/zone.rs:354
18412024-08-29T15:18:17.232ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService23'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::18'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_1b55399c-9dbb-4d32-bd51-426a7996ea24/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::18'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='6a3faa12-4d1f-435b-b060-46272624f3fb'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18422024-08-29T15:18:17.232ZINFOSledAgent (ServiceManager): Booting oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
18432024-08-29T15:18:17.241ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService16'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::16'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_775439b8-d6ea-4b72-9708-a30c4c7c6d6f/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::16'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='9a3e84dc-e0c8-4d3e-9c8f-1b5a823c1471'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18442024-08-29T15:18:17.241ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService21'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::19'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_aad99d77-5c0c-4ae7-8cc1-ced403d6b98a/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::19'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='5b382486-72e3-4c57-83d3-2d93598fd8a6'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18452024-08-29T15:18:17.241ZINFOSledAgent (ServiceManager): Booting oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
18462024-08-29T15:18:17.248ZINFOSledAgent (ServiceManager): Booting oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
18472024-08-29T15:18:17.256ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService19'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::f'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::f'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18482024-08-29T15:18:17.256ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
18492024-08-29T15:18:17.276ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService14'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::17'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_e4f77dae-534d-4b73-b38a-890d525d3747/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::17'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='60622c6b-cb7b-4511-b42d-0e97a87c6a92'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18502024-08-29T15:18:17.277ZINFOSledAgent (ServiceManager): Booting oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
18512024-08-29T15:18:17.286ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService12'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::12'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_05effe78-0c08-40b4-916b-cde344744ec3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::12'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='e24813ac-ee27-4b93-942d-ffa2b0053022'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18522024-08-29T15:18:17.286ZINFOSledAgent (ServiceManager): Booting oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
18532024-08-29T15:18:17.316ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService15'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::15'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_6eaf0007-0aa9-44b6-8bac-b01569f4c6a3/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::15'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='e3fe6845-0be5-4cb9-a51f-9e0cb2d4a4de'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18542024-08-29T15:18:17.316ZINFOSledAgent (ServiceManager): Booting oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
18552024-08-29T15:18:17.324ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService13'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::14'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_4fbbfff0-e175-4404-8cd9-acfc358b7140/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::14'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='2e0b2dcb-d65c-40bb-8422-47dd39ef82d7'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18562024-08-29T15:18:17.325ZINFOSledAgent (ServiceManager): Booting oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
18572024-08-29T15:18:17.407ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService17'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::10'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::10'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18582024-08-29T15:18:17.408ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
18592024-08-29T15:18:17.434ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService22'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::1a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_f4912882-4c2a-4c0d-b986-5cbff960587e/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1a'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='816f8166-ed77-4cfe-82cd-81b0533719ca'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
18602024-08-29T15:18:17.434ZINFOSledAgent (ServiceManager): Booting oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
18612024-08-29T15:18:17.968ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 46.816698ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
18622024-08-29T15:18:17.978ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 55.306726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
18632024-08-29T15:18:17.985ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 59.343624ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
18642024-08-29T15:18:17.994ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 74.441807ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
18652024-08-29T15:18:18.003ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 36.737529ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
18662024-08-29T15:18:18.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 41.361971ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
18672024-08-29T15:18:18.022ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 38.788008ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
18682024-08-29T15:18:18.031ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 62.583881ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
18692024-08-29T15:18:18.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 65.855994ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
18702024-08-29T15:18:18.128ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 65.992149ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
18712024-08-29T15:18:18.143ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 92.683917ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
18722024-08-29T15:18:18.164ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 73.330994ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
18732024-08-29T15:18:18.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 107.856286ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
18742024-08-29T15:18:18.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 133.988223ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
18752024-08-29T15:18:18.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 56.846848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
18762024-08-29T15:18:18.241ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 130.388026ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
18772024-08-29T15:18:18.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 145.987516ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
18782024-08-29T15:18:18.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 64.879527ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
18792024-08-29T15:18:18.298ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 125.86292ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
18802024-08-29T15:18:18.318ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 227.541131ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
18812024-08-29T15:18:18.338ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 74.662678ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
18822024-08-29T15:18:18.356ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 130.755325ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
18832024-08-29T15:18:18.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 112.927229ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
18842024-08-29T15:18:18.456ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 253.488843ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
18852024-08-29T15:18:18.504ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 176.142147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
18862024-08-29T15:18:18.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 150.548514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
18872024-08-29T15:18:18.572ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 154.880614ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
18882024-08-29T15:18:19.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 259.251154ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
18892024-08-29T15:18:19.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 130.656486ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
18902024-08-29T15:18:19.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 369.990918ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
18912024-08-29T15:18:19.284ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 169.904625ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
18922024-08-29T15:18:19.300ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 356.966387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
18932024-08-29T15:18:19.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 312.088385ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
18942024-08-29T15:18:19.359ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 525.410368ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
18952024-08-29T15:18:19.435ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 522.532045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
18962024-08-29T15:18:19.448ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 269.685854ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
18972024-08-29T15:18:19.461ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 538.561535ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
18982024-08-29T15:18:19.475ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 344.845282ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
18992024-08-29T15:18:19.490ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 570.855057ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
19002024-08-29T15:18:19.503ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 588.827349ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19012024-08-29T15:18:19.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.089299362s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
19022024-08-29T15:18:19.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 564.127574ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19032024-08-29T15:18:19.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 684.75022ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19042024-08-29T15:18:19.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 795.957415ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
19052024-08-29T15:18:19.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.027870295s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
19062024-08-29T15:18:19.901ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 729.602273ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19072024-08-29T15:18:19.974ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 663.056813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
19082024-08-29T15:18:20.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 908.537222ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
19092024-08-29T15:18:20.078ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 421.065455ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
19102024-08-29T15:18:20.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 954.363829ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19112024-08-29T15:18:20.267ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 649.718677ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19122024-08-29T15:18:20.416ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 517.271149ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19132024-08-29T15:18:20.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.295878595s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
19142024-08-29T15:18:20.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.123765931s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
19152024-08-29T15:18:20.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.090132658s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19162024-08-29T15:18:20.658ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 687.97612ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
19172024-08-29T15:18:20.759ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 871.961011ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
19182024-08-29T15:18:20.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.032669234s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
19192024-08-29T15:18:20.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 990.52918ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19202024-08-29T15:18:20.942ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.2715803s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
19212024-08-29T15:18:20.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 732.530321ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19222024-08-29T15:18:21.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.296915109s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19232024-08-29T15:18:21.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.326780213s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
19242024-08-29T15:18:21.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.428080366s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
19252024-08-29T15:18:21.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.111849814s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19262024-08-29T15:18:21.715ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.085245366s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
19272024-08-29T15:18:21.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 798.045154ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19282024-08-29T15:18:21.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.048101804s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
19292024-08-29T15:18:21.930ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.008753356s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
19302024-08-29T15:18:21.941ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 557.324518ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19312024-08-29T15:18:22.083ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService10'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::d'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value='776ef919-ac0a-4b14-9dfe-deac75d16696'/> <propval type="astring" name="address" value='[fd00:1122:3344:101::d]:12223'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19322024-08-29T15:18:22.084ZINFOSledAgent (ServiceManager): Booting oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19332024-08-29T15:18:22.089ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService18'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::11'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::11'/> <propval type="astring" name="listen_port" value='17000'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19342024-08-29T15:18:22.089ZINFOSledAgent (ServiceManager): Booting oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19352024-08-29T15:18:22.132ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService20'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::13'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value='oxp_866d818f-82bb-444c-992a-1351dec91efa/crucible'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::13'/> <propval type="astring" name="listen_port" value='32345'/> <propval type="astring" name="uuid" value='0de3f60b-6c72-4697-9744-e3096340231e'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19362024-08-29T15:18:22.132ZINFOSledAgent (ServiceManager): Booting oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19372024-08-29T15:18:22.224ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 881.387514ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
19382024-08-29T15:18:22.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 58.130941ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19392024-08-29T15:18:22.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 55.117997ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19402024-08-29T15:18:22.282ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 33.628622ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19412024-08-29T15:18:22.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 109.756943ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19422024-08-29T15:18:22.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 84.134014ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19432024-08-29T15:18:22.329ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 56.793724ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19442024-08-29T15:18:22.386ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 854.054417ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19452024-08-29T15:18:22.393ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 197.213148ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19462024-08-29T15:18:22.417ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 234.249542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19472024-08-29T15:18:22.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 288.819932ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19482024-08-29T15:18:22.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.290480862s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19492024-08-29T15:18:22.560ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.014896419s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19502024-08-29T15:18:22.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 582.086866ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19512024-08-29T15:18:22.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 427.713004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19522024-08-29T15:18:22.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 301.016456ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19532024-08-29T15:18:22.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.192368442s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
19542024-08-29T15:18:22.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.030465654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
19552024-08-29T15:18:22.822ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.148512474s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19562024-08-29T15:18:22.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.342263775s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
19572024-08-29T15:18:23.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.249272872s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
19582024-08-29T15:18:23.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.051938444s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19592024-08-29T15:18:23.179ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 960.250499ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
19602024-08-29T15:18:23.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 405.014718ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19612024-08-29T15:18:23.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.170418475s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19622024-08-29T15:18:23.220ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 630.636346ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
19632024-08-29T15:18:23.251ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 586.738268ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19642024-08-29T15:18:23.587ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 512.080852ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19652024-08-29T15:18:23.618ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.135037394s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19662024-08-29T15:18:23.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.002254373s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19672024-08-29T15:18:23.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 694.413627ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19682024-08-29T15:18:23.862ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.298578566s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
19692024-08-29T15:18:23.872ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.013209966s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
19702024-08-29T15:18:23.982ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.47085188s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19712024-08-29T15:18:23.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.273251833s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
19722024-08-29T15:18:24.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 797.864455ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19732024-08-29T15:18:24.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.487813477s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
19742024-08-29T15:18:24.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.369350518s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
19752024-08-29T15:18:24.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.441149841s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
19762024-08-29T15:18:24.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.436470204s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
19772024-08-29T15:18:24.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.220353576s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
19782024-08-29T15:18:24.437ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService11'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::e'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::e'/> <propval type="astring" name="listen_port" value='8123'/> <propval type="astring" name="store" value='/data'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:1::1" /> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="clickhouse_address" value='[fd00:1122:3344:101::e]:8123'/> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::e]:8888'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19792024-08-29T15:18:24.437ZINFOSledAgent (ServiceManager): Booting oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
19802024-08-29T15:18:24.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 854.60059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19812024-08-29T15:18:24.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 41.9639ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
19822024-08-29T15:18:24.705ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 64.946891ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
19832024-08-29T15:18:24.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 790.725381ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
19842024-08-29T15:18:24.777ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 268.653934ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
19852024-08-29T15:18:24.824ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.271937257s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
19862024-08-29T15:18:24.896ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 961.275855ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
19872024-08-29T15:18:24.920ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 931.617668ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
19882024-08-29T15:18:25.043ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService26'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::c'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte3'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.7'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
19892024-08-29T15:18:25.043ZINFOSledAgent (ServiceManager): Booting oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
19902024-08-29T15:18:25.079ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 398.654124ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
19912024-08-29T15:18:25.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.188779023s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
19922024-08-29T15:18:25.201ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 39.280903ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
19932024-08-29T15:18:25.248ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 114.729976ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
19942024-08-29T15:18:25.275ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.399584977s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
19952024-08-29T15:18:25.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 230.261551ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
19962024-08-29T15:18:25.423ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.173469952s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
19972024-08-29T15:18:25.462ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.031089492s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
19982024-08-29T15:18:25.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 677.205211ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
19992024-08-29T15:18:25.530ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 623.988741ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
20002024-08-29T15:18:25.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 928.229648ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
20012024-08-29T15:18:25.616ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 223.981667ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20022024-08-29T15:18:25.624ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 660.284263ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
20032024-08-29T15:18:25.633ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.265419441s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
20042024-08-29T15:18:25.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.080189139s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
20052024-08-29T15:18:25.741ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.058446855s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
20062024-08-29T15:18:25.848ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 1.105325745s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20072024-08-29T15:18:25.861ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.290259779s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
20082024-08-29T15:18:25.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.037993654s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
20092024-08-29T15:18:26.105ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.417651145s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
20102024-08-29T15:18:26.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 552.8967ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
20112024-08-29T15:18:26.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 1.285648097s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
20122024-08-29T15:18:26.292ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.316655909s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
20132024-08-29T15:18:26.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.430636771s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
20142024-08-29T15:18:26.499ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.343259963s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
20152024-08-29T15:18:26.505ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.372978237s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
20162024-08-29T15:18:26.604ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.243954378s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
20172024-08-29T15:18:26.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 783.931295ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
20182024-08-29T15:18:26.737ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 802.429831ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
20192024-08-29T15:18:26.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.466729611s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
20202024-08-29T15:18:26.971ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.250616229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
20212024-08-29T15:18:27.017ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 673.753107ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20222024-08-29T15:18:27.059ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 622.999602ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
20232024-08-29T15:18:27.080ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.133693141s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
20242024-08-29T15:18:27.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.092473456s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
20252024-08-29T15:18:27.471ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 1.244053515s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
20262024-08-29T15:18:27.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.4603862s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
20272024-08-29T15:18:27.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 774.554004ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
20282024-08-29T15:18:27.549ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.060318818s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
20292024-08-29T15:18:27.619ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 839.752662ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
20302024-08-29T15:18:27.691ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.481622778s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
20312024-08-29T15:18:27.700ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 721.029247ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20322024-08-29T15:18:27.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.115102146s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
20332024-08-29T15:18:27.852ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 787.687396ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
20342024-08-29T15:18:27.852ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService28'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::9'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte5'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::9]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.6:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20352024-08-29T15:18:27.860ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.328601104s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
20362024-08-29T15:18:27.860ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20372024-08-29T15:18:27.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 802.342418ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
20382024-08-29T15:18:28.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 45.62892ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20392024-08-29T15:18:28.148ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 67.131458ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20402024-08-29T15:18:28.223ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 993.622245ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
20412024-08-29T15:18:28.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 103.631148ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20422024-08-29T15:18:28.242ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 693.094333ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
20432024-08-29T15:18:28.268ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.477205389s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
20442024-08-29T15:18:28.308ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 945.095347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
20452024-08-29T15:18:28.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 814.925387ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
20462024-08-29T15:18:28.344ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 209.981133ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20472024-08-29T15:18:28.432ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 1.250330979s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20482024-08-29T15:18:28.467ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.467535277s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
20492024-08-29T15:18:28.520ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService25'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::b'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte2'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.6'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20502024-08-29T15:18:28.520ZINFOSledAgent (ServiceManager): Booting oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20512024-08-29T15:18:28.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.05822442s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20522024-08-29T15:18:28.589ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService24'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::a'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte1'/> <propval type="astring" name="gateway" value='172.30.2.1'/> <propval type="astring" name="ip" value='172.30.2.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
20532024-08-29T15:18:28.589ZINFOSledAgent (ServiceManager): Booting oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20542024-08-29T15:18:28.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.275503307s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
20552024-08-29T15:18:28.649ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.278335303s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
20562024-08-29T15:18:28.677ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 71.579584ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20572024-08-29T15:18:28.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 661.789701ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
20582024-08-29T15:18:28.724ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 691.771755ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
20592024-08-29T15:18:28.731ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 62.446386ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20602024-08-29T15:18:28.757ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 79.764238ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20612024-08-29T15:18:28.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 124.859139ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20622024-08-29T15:18:28.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 164.864621ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20632024-08-29T15:18:28.934ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 870.830216ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
20642024-08-29T15:18:28.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 189.649332ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20652024-08-29T15:18:28.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.001578033s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
20662024-08-29T15:18:28.958ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.382248918s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
20672024-08-29T15:18:29.023ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 399.80287ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20682024-08-29T15:18:29.142ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.39744566s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
20692024-08-29T15:18:29.150ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 273.336198ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20702024-08-29T15:18:29.181ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.295276915s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
20712024-08-29T15:18:29.197ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 507.695939ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
20722024-08-29T15:18:29.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 805.580596ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
20732024-08-29T15:18:29.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 773.050091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
20742024-08-29T15:18:29.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 516.368702ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
20752024-08-29T15:18:29.425ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 787.824721ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
20762024-08-29T15:18:29.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 427.32072ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20772024-08-29T15:18:29.442ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 763.168602ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20782024-08-29T15:18:29.632ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.198365311s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
20792024-08-29T15:18:29.690ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 691.10591ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20802024-08-29T15:18:29.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 816.327819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
20812024-08-29T15:18:29.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 872.999694ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
20822024-08-29T15:18:29.812ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.481297446s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
20832024-08-29T15:18:29.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 789.20595ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
20842024-08-29T15:18:29.904ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 616.905124ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
20852024-08-29T15:18:29.913ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.309904817s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
20862024-08-29T15:18:29.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 538.001339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
20872024-08-29T15:18:29.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 705.693102ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
20882024-08-29T15:18:30.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.408486162s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
20892024-08-29T15:18:30.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.336759285s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
20902024-08-29T15:18:30.279ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.310162044s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
20912024-08-29T15:18:30.296ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 797.587758ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
20922024-08-29T15:18:30.451ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 996.309165ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
20932024-08-29T15:18:30.514ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 670.039673ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
20942024-08-29T15:18:30.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.331899981s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
20952024-08-29T15:18:30.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.472710861s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
20962024-08-29T15:18:30.600ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 650.819051ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
20972024-08-29T15:18:30.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 926.685082ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
20982024-08-29T15:18:30.623ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 575.485578ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
20992024-08-29T15:18:30.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.371066546s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
21002024-08-29T15:18:30.650ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.012138009s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
21012024-08-29T15:18:30.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.039800083s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21022024-08-29T15:18:30.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 1.401338815s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
21032024-08-29T15:18:30.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.263078595s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
21042024-08-29T15:18:31.104ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 978.925623ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
21052024-08-29T15:18:31.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 786.026325ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
21062024-08-29T15:18:31.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.264320338s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
21072024-08-29T15:18:31.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 791.956835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
21082024-08-29T15:18:31.260ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 731.080144ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
21092024-08-29T15:18:31.302ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.474581323s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
21102024-08-29T15:18:31.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 746.776577ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
21112024-08-29T15:18:31.543ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 788.440123ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
21122024-08-29T15:18:31.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 917.23303ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
21132024-08-29T15:18:31.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.238785755s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
21142024-08-29T15:18:31.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.282520363s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
21152024-08-29T15:18:31.670ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 802.392802ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
21162024-08-29T15:18:31.712ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 653.674348ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21172024-08-29T15:18:31.905ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 665.270542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
21182024-08-29T15:18:31.989ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 1.166652413s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
21192024-08-29T15:18:31.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.149171317s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
21202024-08-29T15:18:32.014ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.194810676s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
21212024-08-29T15:18:32.032ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.26809989s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
21222024-08-29T15:18:32.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.072268994s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
21232024-08-29T15:18:32.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 1.151698218s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
21242024-08-29T15:18:32.092ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 518.847861ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
21252024-08-29T15:18:32.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.352098494s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
21262024-08-29T15:18:32.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 1.054341353s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
21272024-08-29T15:18:32.342ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.130928682s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
21282024-08-29T15:18:32.375ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.308834416s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21292024-08-29T15:18:32.479ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.070356946s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
21302024-08-29T15:18:32.487ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 695.740134ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
21312024-08-29T15:18:32.495ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 872.600447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
21322024-08-29T15:18:32.580ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 709.368213ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
21332024-08-29T15:18:32.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 1.111869242s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
21342024-08-29T15:18:32.786ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 564.889696ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
21352024-08-29T15:18:32.837ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.32840225s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
21362024-08-29T15:18:32.888ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.189501645s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
21372024-08-29T15:18:33.152ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 570.450773ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
21382024-08-29T15:18:33.160ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 723.138282ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
21392024-08-29T15:18:33.169ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 769.569366ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
21402024-08-29T15:18:33.193ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 945.668447ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
21412024-08-29T15:18:33.217ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.138212207s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
21422024-08-29T15:18:33.246ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 1.44944176s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
21432024-08-29T15:18:33.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 763.524781ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
21442024-08-29T15:18:33.299ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 523.557746ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
21452024-08-29T15:18:33.309ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 940.833916ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
21462024-08-29T15:18:33.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.164609905s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
21472024-08-29T15:18:33.376ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.486055919s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
21482024-08-29T15:18:33.474ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 684.295062ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
21492024-08-29T15:18:33.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 517.145203ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
21502024-08-29T15:18:33.559ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 717.95703ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
21512024-08-29T15:18:33.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.166206225s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21522024-08-29T15:18:33.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.415705307s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
21532024-08-29T15:18:33.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 899.247461ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
21542024-08-29T15:18:33.833ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 693.795283ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
21552024-08-29T15:18:33.894ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 534.609113ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
21562024-08-29T15:18:33.951ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 1.131396834s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
21572024-08-29T15:18:34.011ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.386271929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
21582024-08-29T15:18:34.053ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 975.91013ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
21592024-08-29T15:18:34.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 951.876098ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
21602024-08-29T15:18:34.151ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 834.171582ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
21612024-08-29T15:18:34.172ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.292167895s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
21622024-08-29T15:18:34.183ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.409421229s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
21632024-08-29T15:18:34.263ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.278468532s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
21642024-08-29T15:18:34.291ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 980.276542ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
21652024-08-29T15:18:34.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.111478083s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
21662024-08-29T15:18:34.447ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.037652572s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
21672024-08-29T15:18:34.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.423324861s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
21682024-08-29T15:18:34.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 605.295821ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
21692024-08-29T15:18:34.563ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlService27'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="static_addr" value='fd00:1122:3344:101::8'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value='opte4'/> <propval type="astring" name="gateway" value='172.30.1.1'/> <propval type="astring" name="ip" value='172.30.1.5'/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value='[fd00:1122:3344:101::8]:5353'/> <propval type="astring" name="dns_address" value='172.30.1.5:53'/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
21702024-08-29T15:18:34.563ZINFOSledAgent (ServiceManager): Booting oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974 zone file = illumos-utils/src/running_zone.rs:488 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
21712024-08-29T15:18:34.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 1.040103625s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
21722024-08-29T15:18:34.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 801.74006ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
21732024-08-29T15:18:34.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 523.464545ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21742024-08-29T15:18:34.911ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 869.636174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
21752024-08-29T15:18:34.929ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 35.577354ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
21762024-08-29T15:18:34.999ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 104.72522ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
21772024-08-29T15:18:35.027ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.031831298s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
21782024-08-29T15:18:35.064ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 960.424096ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
21792024-08-29T15:18:35.097ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 921.532271ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
21802024-08-29T15:18:35.116ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 1.438010887s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
21812024-08-29T15:18:35.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 257.498791ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
21822024-08-29T15:18:35.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.126684049s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
21832024-08-29T15:18:35.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.180109364s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
21842024-08-29T15:18:35.313ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 718.735835ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
21852024-08-29T15:18:35.421ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 499.157796ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
21862024-08-29T15:18:35.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 576.72574ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21872024-08-29T15:18:35.449ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 620.592713ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
21882024-08-29T15:18:35.496ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.353969226s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
21892024-08-29T15:18:35.541ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 966.932561ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
21902024-08-29T15:18:35.557ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.111443217s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
21912024-08-29T15:18:35.579ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 878.141161ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
21922024-08-29T15:18:35.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.062386366s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
21932024-08-29T15:18:35.620ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.451611893s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
21942024-08-29T15:18:35.716ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 1.373465479s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
21952024-08-29T15:18:35.809ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 994.540288ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
21962024-08-29T15:18:35.939ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 1.005485038s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
21972024-08-29T15:18:35.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.365459921s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
21982024-08-29T15:18:36.035ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.184172072s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
21992024-08-29T15:18:36.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 1.0040339s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22002024-08-29T15:18:36.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 968.985676ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
22012024-08-29T15:18:36.106ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 743.862578ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22022024-08-29T15:18:36.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.33134984s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
22032024-08-29T15:18:36.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 554.401009ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
22042024-08-29T15:18:36.308ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.300857143s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
22052024-08-29T15:18:36.389ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 809.255616ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
22062024-08-29T15:18:36.468ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 1.428451816s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
22072024-08-29T15:18:36.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 998.006732ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22082024-08-29T15:18:36.564ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 957.709301ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
22092024-08-29T15:18:36.674ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.399847802s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
22102024-08-29T15:18:36.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.263927929s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
22112024-08-29T15:18:36.701ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.194122659s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
22122024-08-29T15:18:36.814ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.212359833s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
22132024-08-29T15:18:36.859ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 645.248562ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
22142024-08-29T15:18:36.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 531.830802ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22152024-08-29T15:18:36.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 1.257659454s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22162024-08-29T15:18:37.070ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 845.881106ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
22172024-08-29T15:18:37.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 725.564604ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22182024-08-29T15:18:37.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.379725619s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
22192024-08-29T15:18:37.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 1.304450851s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
22202024-08-29T15:18:37.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.099151132s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
22212024-08-29T15:18:37.233ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 827.435009ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
22222024-08-29T15:18:37.365ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.246641122s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
22232024-08-29T15:18:37.411ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 904.173439ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22242024-08-29T15:18:37.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 605.043174ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
22252024-08-29T15:18:37.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 583.285851ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
22262024-08-29T15:18:37.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 734.813819ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22272024-08-29T15:18:37.536ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 689.95374ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
22282024-08-29T15:18:37.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 837.363616ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
22292024-08-29T15:18:37.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 555.572543ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22302024-08-29T15:18:37.909ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 1.181497297s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
22312024-08-29T15:18:37.924ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 844.949517ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
22322024-08-29T15:18:37.936ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 991.951444ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
22332024-08-29T15:18:37.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 785.643425ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
22342024-08-29T15:18:38.043ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.054281508s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
22352024-08-29T15:18:38.072ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.400797075s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
22362024-08-29T15:18:38.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.104683472s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
22372024-08-29T15:18:38.094ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 892.105179ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
22382024-08-29T15:18:38.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 1.492145196s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
22392024-08-29T15:18:38.225ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 790.797962ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22402024-08-29T15:18:38.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 714.85077ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
22412024-08-29T15:18:38.270ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 569.370987ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22422024-08-29T15:18:38.321ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 1.451487821s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
22432024-08-29T15:18:38.331ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 603.998658ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22442024-08-29T15:18:38.385ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 823.480687ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22452024-08-29T15:18:38.418ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 1.237418176s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
22462024-08-29T15:18:38.470ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 796.833059ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
22472024-08-29T15:18:38.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.465850897s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
22482024-08-29T15:18:38.625ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 521.25147ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
22492024-08-29T15:18:38.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 1.380093061s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
22502024-08-29T15:18:38.783ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 1.126848613s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
22512024-08-29T15:18:38.855ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 805.277329ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22522024-08-29T15:18:38.943ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 585.391726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
22532024-08-29T15:18:38.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 929.686025ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22542024-08-29T15:18:38.967ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 1.335993963s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
22552024-08-29T15:18:39.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.002000612s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
22562024-08-29T15:18:39.030ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 931.978981ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22572024-08-29T15:18:39.107ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 968.519692ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
22582024-08-29T15:18:39.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 1.078352996s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
22592024-08-29T15:18:39.162ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691") failed: Property not found. retry in 1.283397775s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
22602024-08-29T15:18:39.203ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 859.829775ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
22612024-08-29T15:18:39.221ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 1.394577475s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22622024-08-29T15:18:39.286ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.486632861s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
22632024-08-29T15:18:39.489ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696") failed: Property not found. retry in 1.422946245s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
22642024-08-29T15:18:39.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247") failed: Property not found. retry in 1.337393823s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
22652024-08-29T15:18:39.622ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 880.624123ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
22662024-08-29T15:18:39.679ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5") failed: Property not found. retry in 1.280951137s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
22672024-08-29T15:18:39.695ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 538.634473ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22682024-08-29T15:18:39.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 564.125854ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
22692024-08-29T15:18:39.912ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 637.460165ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22702024-08-29T15:18:39.922ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9") failed: Property not found. retry in 1.495738969s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
22712024-08-29T15:18:39.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 556.226463ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
22722024-08-29T15:18:39.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 1.118577774s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22732024-08-29T15:18:40.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3") failed: Property not found. retry in 1.252190516s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
22742024-08-29T15:18:40.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873") failed: Property not found. retry in 1.049137211s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
22752024-08-29T15:18:40.091ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2") failed: Property not found. retry in 922.570576ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
22762024-08-29T15:18:40.165ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c") failed: Property not found. retry in 847.209659ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
22772024-08-29T15:18:40.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a") failed: Property not found. retry in 707.070354ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
22782024-08-29T15:18:40.252ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851") failed: Property not found. retry in 1.004740284s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22792024-08-29T15:18:40.323ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e") failed: Property not found. retry in 727.118689ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
22802024-08-29T15:18:40.370ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9") failed: Property not found. retry in 586.244787ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
22812024-08-29T15:18:40.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9") failed: Property not found. retry in 639.228228ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
22822024-08-29T15:18:40.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348") failed: Property not found. retry in 1.31170411s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
22832024-08-29T15:18:40.575ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03") failed: Property not found. retry in 1.027244931s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
22842024-08-29T15:18:40.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0") failed: Property not found. retry in 1.45098224s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22852024-08-29T15:18:40.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932") failed: Property not found. retry in 1.127826397s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
22862024-08-29T15:18:41.332ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 932.506691ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22872024-08-29T15:18:42.326ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 663.224774ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22882024-08-29T15:18:42.508ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35625
22892024-08-29T15:18:42.512ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35625 req_id = 9ab68e08-5ea9-4d5b-9e66-68ffae55d766 response_code = 200 uri = /baseboard
22902024-08-29T15:18:43.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974") failed: Property not found. retry in 1.283719925s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
22912024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_oximeter_776ef919-ac0a-4b14-9dfe-deac75d16696
22922024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_nexus_a66efbaf-5f9a-40ef-9401-e26ed785dbe9
22932024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_external_dns_49d104a1-1335-4930-9b9e-1ec4a3ca16f9
22942024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_nexus_d36d5399-1e8d-42d4-b1c6-82ab39a196e5
22952024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_clickhouse_709ff80f-750f-45c6-b668-97969f356dc0
22962024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_5b880f4d-eaa0-4b77-919b-817d6a980247
22972024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_5a83427f-50d0-47ad-8765-fad7f4c19873
22982024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_d98c3f2a-4c9c-4655-9407-3839f53cf851
22992024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_f389808b-a883-46ea-a396-af6431b5b6e3
23002024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_23eb92fe-33fd-4e95-a819-b3349525e67a
23012024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_nexus_a752201f-8e19-45ac-9901-65317e43fc2e
23022024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_pantry_45d4b7eb-c738-4794-9125-01d0ae07feb2
23032024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_pantry_bbf9c6ad-fa23-470f-b608-67105cf521a9
23042024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_pantry_4c2f62ac-cb40-467c-ba9f-de7aa05f3691
23052024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_aa3cf0ac-1c68-4e1c-a127-490862a19932
23062024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_9becf607-a424-4af6-bd2e-178c196b8348
23072024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_3da8a36e-1727-4d21-9b85-8be498e95b8c
23082024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_crucible_57f24d92-52ba-4db7-a1b1-c2a25e015a03
23092024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:3250 zone = oxz_external_dns_45f571be-936e-48ca-b02e-6bbc2b2f1974
23102024-08-29T15:18:44.334ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
23112024-08-29T15:18:44.336ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
23122024-08-29T15:18:44.337ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 31863706 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:65026 req_id = 757b4cd9-d074-49c0-9f45-b5efba9434dd response_code = 204 uri = /omicron-zones
23132024-08-29T15:18:44.342ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1286
23142024-08-29T15:18:44.342ZINFOSledAgent (RSS): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
23152024-08-29T15:18:44.342ZINFOSledAgent (RSS): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
23162024-08-29T15:18:44.346ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:687
23172024-08-29T15:18:44.346ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::a]:12221 file = sled-agent/src/rack_setup/service.rs:700
23182024-08-29T15:18:44.379ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], lldp: None, port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), local_pref: None, nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:867
23192024-08-29T15:18:44.379ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23202024-08-29T15:18:44.455ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23212024-08-29T15:18:44.629ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23222024-08-29T15:18:44.811ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23232024-08-29T15:18:44.999ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23242024-08-29T15:18:45.158ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23252024-08-29T15:18:45.327ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23262024-08-29T15:18:45.526ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23272024-08-29T15:18:45.774ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23282024-08-29T15:18:46.035ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23292024-08-29T15:18:46.702ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23302024-08-29T15:18:47.460ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23312024-08-29T15:18:47.998ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23322024-08-29T15:18:48.819ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23332024-08-29T15:18:49.999ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23342024-08-29T15:18:51.668ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23352024-08-29T15:18:52.994ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23362024-08-29T15:18:54.086ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23372024-08-29T15:18:54.086ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23382024-08-29T15:18:54.480ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23392024-08-29T15:18:54.485ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23402024-08-29T15:18:55.089ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23412024-08-29T15:18:56.092ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23422024-08-29T15:18:56.479ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23432024-08-29T15:18:56.495ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23442024-08-29T15:18:57.096ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23452024-08-29T15:18:57.921ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23462024-08-29T15:18:58.100ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23472024-08-29T15:18:58.479ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23482024-08-29T15:18:59.104ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23492024-08-29T15:18:59.689ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::a]:12221/racks/0c368d89-20a1-4c09-9eea-581bb60b3770/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:929
23502024-08-29T15:19:00.107ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23512024-08-29T15:19:00.479ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/nexus.rs:571 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23522024-08-29T15:19:01.114ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/b9c40332-a231-42db-adf0-b267d663a311): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/probe_manager.rs:230 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23532024-08-29T15:19:03.120ZINFOSledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "1ffcd8ea-e615-45b9-8415-54f197693d3a", "content-length": "173", "date": "Thu, 29 Aug 2024 15:19:03 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"0c368d89-20a1-4c09-9eea-581bb60b3770\"", request_id: "1ffcd8ea-e615-45b9-8415-54f197693d3a" } file = sled-agent/src/rack_setup/service.rs:929
23542024-08-29T15:19:04.577ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:544 new_generation = 1 old_generation = 1 sled_id = b9c40332-a231-42db-adf0-b267d663a311
23552024-08-29T15:19:04.667ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33841
23562024-08-29T15:19:04.668ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23572024-08-29T15:19:04.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 78819 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33841 req_id = 243235e1-62f9-4de7-99d7-a48947be1256 response_code = 200 uri = /inventory
23582024-08-29T15:19:04.746ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23592024-08-29T15:19:04.746ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
23602024-08-29T15:19:04.747ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
23612024-08-29T15:19:04.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1105 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33841 req_id = b995e923-f7aa-4991-8ef0-61249f2945b2 response_code = 200 uri = /omicron-zones
23622024-08-29T15:19:09.550ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41070
23632024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23642024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23652024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23662024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23672024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23682024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
23692024-08-29T15:19:09.550ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23702024-08-29T15:19:09.550ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 461 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:41070 req_id = dfb5d0d9-7639-4919-bb9c-cec0f2f16e0f response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23712024-08-29T15:19:09.552ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:939
23722024-08-29T15:19:09.907ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47586
23732024-08-29T15:19:09.908ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23742024-08-29T15:19:10.004ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96905 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47586 req_id = 19631bfc-8ef1-4cc8-8253-6c3297b0c25a response_code = 200 uri = /inventory
23752024-08-29T15:19:10.005ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
23762024-08-29T15:19:10.005ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/config/omicron-zones.json file = common/src/ledger.rs:177
23772024-08-29T15:19:10.005ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/config/omicron-zones.json file = common/src/ledger.rs:177
23782024-08-29T15:19:10.006ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1030 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47586 req_id = 8334647d-1d5b-49f6-80ff-1388bff3c277 response_code = 200 uri = /omicron-zones
23792024-08-29T15:19:10.125ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:42797
23802024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23812024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23822024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23832024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23842024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23852024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
23862024-08-29T15:19:10.126ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23872024-08-29T15:19:10.126ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 996 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:42797 req_id = c7c6d734-5163-413a-b7ee-4e77785b8c61 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23882024-08-29T15:19:10.297ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49176
23892024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23902024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23912024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23922024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
23932024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23942024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
23952024-08-29T15:19:10.298ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
23962024-08-29T15:19:10.298ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 297 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:49176 req_id = b498b695-e8ce-4cd4-9c4d-dbe6915912b8 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23972024-08-29T15:19:10.348ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60529
23982024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
23992024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24002024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24012024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
24022024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24032024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
24042024-08-29T15:19:10.348ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
24052024-08-29T15:19:10.348ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 199 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:60529 req_id = d240b26f-713f-44c4-bfbe-f689a03e0516 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
24062024-08-29T15:19:12.551ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64403
24072024-08-29T15:19:12.551ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 31 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64403 req_id = 1f27779f-7d3e-4bba-8ec1-ae6901343175 response_code = 200 uri = /baseboard
24082024-08-29T15:19:18.472ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37831
24092024-08-29T15:19:18.472ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37831 req_id = bd38a7d8-f596-4990-9b98-466226002c98 response_code = 200 uri = /vpc-routes
24102024-08-29T15:19:18.551ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 708 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:37831 req_id = bc62cc65-806f-4d54-a23e-38a4e600bf0c response_code = 204 uri = /vpc-routes
24112024-08-29T15:19:29.435ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
24122024-08-29T15:19:33.779ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36529
24132024-08-29T15:19:33.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36529 req_id = 1bb68e8e-508e-4691-a8f7-b2cfde10d094 response_code = 200 uri = /vpc-routes
24142024-08-29T15:19:33.804ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:36529 req_id = ae2f2eb1-0fa4-4eba-bfc1-a7138b22c71a response_code = 204 uri = /vpc-routes
24152024-08-29T15:19:33.857ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53958
24162024-08-29T15:19:33.857ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53958 req_id = efb51599-bce4-4dac-959d-72db079bc5f9 response_code = 200 uri = /vpc-routes
24172024-08-29T15:19:33.878ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 203 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53958 req_id = e137a4ca-d34e-45f1-8dc9-e05aff71dcb9 response_code = 204 uri = /vpc-routes
24182024-08-29T15:19:33.914ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58460
24192024-08-29T15:19:33.915ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1052 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58460 req_id = c47ab61a-a5a6-47a1-8afb-51bada1cc923 response_code = 200 uri = /vpc-routes
24202024-08-29T15:19:33.920ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:37885
24212024-08-29T15:19:33.920ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:37885 req_id = 8d544c33-b134-495e-9824-ebb3f08357bb response_code = 200 uri = /v2p
24222024-08-29T15:19:33.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:58460 req_id = 4ae60ceb-c3fd-4dcd-b753-d7d0090dfda2 response_code = 204 uri = /vpc-routes
24232024-08-29T15:19:33.973ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56074
24242024-08-29T15:19:33.973ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24252024-08-29T15:19:34.026ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24262024-08-29T15:19:34.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115540 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:56074 req_id = 65485761-657c-49f0-8dfa-d9d653ebf39b response_code = 204 uri = /switch-ports
24272024-08-29T15:19:34.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39472
24282024-08-29T15:19:34.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59739
24292024-08-29T15:19:34.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 235 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39472 req_id = 979baee9-e9ec-4c3e-8d96-6769dde4099d response_code = 200 uri = /v2p
24302024-08-29T15:19:34.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 302 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56074 req_id = 191582b6-86c2-4629-b352-82b937906e10 response_code = 200 uri = /network-bootstore-config
24312024-08-29T15:19:34.090ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 983 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59739 req_id = 15c78246-2327-4a90-aebd-177a35abd812 response_code = 200 uri = /v2p
24322024-08-29T15:19:34.123ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36790
24332024-08-29T15:19:34.124ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24342024-08-29T15:19:34.164ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24352024-08-29T15:19:34.225ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101537 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36790 req_id = fc8ea60d-2c34-462b-a54f-acf6f4f69e9e response_code = 204 uri = /switch-ports
24362024-08-29T15:19:34.226ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
24372024-08-29T15:19:34.226ZINFOSledAgent (bootstore): Writing ledger to /pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
24382024-08-29T15:19:34.227ZINFOSledAgent (bootstore): Writing ledger to /pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
24392024-08-29T15:19:34.227ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
24402024-08-29T15:19:34.227ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2200 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:56074 req_id = 24f99483-5596-4076-810f-ceb97b060c49 response_code = 204 uri = /network-bootstore-config
24412024-08-29T15:19:34.228ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1929 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36790 req_id = 857b4a37-497e-46c7-b7a2-e47c32afd842 response_code = 200 uri = /network-bootstore-config
24422024-08-29T15:19:34.281ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44066
24432024-08-29T15:19:34.281ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24442024-08-29T15:19:34.315ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24452024-08-29T15:19:34.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88627 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:44066 req_id = 2eb6e8aa-c9d7-47f0-a027-6de7958c5892 response_code = 204 uri = /switch-ports
24462024-08-29T15:19:34.370ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 212 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44066 req_id = 9e6f5a31-788f-430c-94f0-f76be59b51e1 response_code = 200 uri = /network-bootstore-config
24472024-08-29T15:19:42.585ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:35102
24482024-08-29T15:19:42.585ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:35102 req_id = fe8ee4fb-8a15-4d4c-b22d-0f84c7cbb787 response_code = 200 uri = /baseboard
24492024-08-29T15:20:03.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48971
24502024-08-29T15:20:03.966ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1260 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48971 req_id = b30b99ba-1a6a-4583-9d6a-3d32105de686 response_code = 200 uri = /vpc-routes
24512024-08-29T15:20:03.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46912
24522024-08-29T15:20:03.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46912 req_id = e0197960-7107-4fd0-b4cd-8ff11134592e response_code = 200 uri = /vpc-routes
24532024-08-29T15:20:04.002ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:46912 req_id = b800aa81-9ea6-4b46-a1ef-dcd9716fbafa response_code = 204 uri = /vpc-routes
24542024-08-29T15:20:04.073ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57276
24552024-08-29T15:20:04.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57276 req_id = 9e1fd0f4-c966-4d80-99d2-51eb3c99a41e response_code = 200 uri = /v2p
24562024-08-29T15:20:04.091ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35579
24572024-08-29T15:20:04.091ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 59 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:35579 req_id = 90882e64-e7f8-470c-bb70-bd39ad9cf4ba response_code = 200 uri = /vpc-routes
24582024-08-29T15:20:04.103ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 969 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35579 req_id = 93202793-60af-46fb-95eb-69dfd33027e4 response_code = 204 uri = /vpc-routes
24592024-08-29T15:20:04.157ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45572
24602024-08-29T15:20:04.157ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45572 req_id = 6f02b2df-9ef4-442c-b649-f10efedb42f7 response_code = 200 uri = /v2p
24612024-08-29T15:20:04.167ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:48971 req_id = ae15e966-e872-48db-8c6a-8c8d1fdc7b33 response_code = 204 uri = /vpc-routes
24622024-08-29T15:20:04.224ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62216
24632024-08-29T15:20:04.224ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 84 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62216 req_id = 4aba0bcc-dd7f-4770-83ad-af288ed452a4 response_code = 200 uri = /v2p
24642024-08-29T15:20:04.330ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54178
24652024-08-29T15:20:04.330ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24662024-08-29T15:20:04.366ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24672024-08-29T15:20:04.419ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 89407 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:54178 req_id = 14998f7b-4bf1-4ab7-b737-b37e2462e94f response_code = 204 uri = /switch-ports
24682024-08-29T15:20:04.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63958
24692024-08-29T15:20:04.419ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:57943
24702024-08-29T15:20:04.420ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 313 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54178 req_id = 06242728-ca67-44fd-b062-117b6bf0672d response_code = 200 uri = /network-bootstore-config
24712024-08-29T15:20:04.420ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24722024-08-29T15:20:04.459ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24732024-08-29T15:20:04.517ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24742024-08-29T15:20:04.517ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97637 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:57943 req_id = d5dc547b-7978-4c62-9f12-78dba0b4742d response_code = 204 uri = /switch-ports
24752024-08-29T15:20:04.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 700 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:57943 req_id = 3dcf6dda-8def-4553-949d-c4b894670b42 response_code = 200 uri = /network-bootstore-config
24762024-08-29T15:20:04.555ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24772024-08-29T15:20:04.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 189659 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63958 req_id = ee563b79-657f-4775-b82a-3c9d7b3b5301 response_code = 204 uri = /switch-ports
24782024-08-29T15:20:04.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 142 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63958 req_id = b8fc0f5d-e4d4-47f2-92a5-96d8bf370b72 response_code = 200 uri = /network-bootstore-config
24792024-08-29T15:20:12.304ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:34899 remote_addr = [fd00:1122:3344:101::d]:51670 sled_id = b9c40332-a231-42db-adf0-b267d663a311
24802024-08-29T15:20:12.358ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 54108 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = f6fd873e-67ca-4cd7-aeeb-d4cc1d56daea response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
24812024-08-29T15:20:12.618ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33843
24822024-08-29T15:20:12.618ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33843 req_id = 29b07778-6d00-49fd-8127-205a9241d6ff response_code = 200 uri = /baseboard
24832024-08-29T15:20:33.805ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61936
24842024-08-29T15:20:33.805ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61936 req_id = 2bb9e2a7-fb48-4931-9170-9816a62cfb18 response_code = 200 uri = /vpc-routes
24852024-08-29T15:20:33.823ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61936 req_id = c7e8e9e5-b77a-4e7c-a236-7125e5caab63 response_code = 204 uri = /vpc-routes
24862024-08-29T15:20:33.827ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33350
24872024-08-29T15:20:33.828ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 53 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33350 req_id = cbc726a1-cc38-4981-8d34-b3b0b6d11f8e response_code = 200 uri = /vpc-routes
24882024-08-29T15:20:33.842ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33350 req_id = 16ed82be-159c-4640-9516-82c0b2ba72af response_code = 204 uri = /vpc-routes
24892024-08-29T15:20:33.913ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63264
24902024-08-29T15:20:33.913ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 60 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63264 req_id = a8cf00d3-e642-4472-8d90-56b5154d2ec1 response_code = 200 uri = /vpc-routes
24912024-08-29T15:20:33.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63264 req_id = 24cc28cd-16d7-4a36-8539-1892e0667d22 response_code = 204 uri = /vpc-routes
24922024-08-29T15:20:33.937ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40353
24932024-08-29T15:20:33.937ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40353 req_id = fc533a6e-3847-425c-94e8-0f937e368914 response_code = 200 uri = /v2p
24942024-08-29T15:20:33.965ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43224
24952024-08-29T15:20:33.965ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
24962024-08-29T15:20:34.008ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
24972024-08-29T15:20:34.072ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 107172 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43224 req_id = cc2cea85-7546-4084-9ab1-ec384ca3dab6 response_code = 204 uri = /switch-ports
24982024-08-29T15:20:34.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:64289
24992024-08-29T15:20:34.072ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52263
25002024-08-29T15:20:34.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:64289 req_id = 83633b0f-a57c-4561-b708-543a136390fb response_code = 200 uri = /v2p
25012024-08-29T15:20:34.073ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 258 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43224 req_id = 336eecb4-b97d-4e11-995a-98fc48a2c3a8 response_code = 200 uri = /network-bootstore-config
25022024-08-29T15:20:34.074ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25032024-08-29T15:20:34.093ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39992
25042024-08-29T15:20:34.094ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39992 req_id = 5a6b384e-193a-4784-bccd-ff55ec4a5db8 response_code = 200 uri = /v2p
25052024-08-29T15:20:34.115ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25062024-08-29T15:20:34.158ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55121
25072024-08-29T15:20:34.176ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25082024-08-29T15:20:34.176ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 103645 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52263 req_id = 4e25449c-a092-4682-a270-e32be622ccfd response_code = 204 uri = /switch-ports
25092024-08-29T15:20:34.181ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 730 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52263 req_id = 2a5aeeb2-722d-44af-b613-862814cc851b response_code = 200 uri = /network-bootstore-config
25102024-08-29T15:20:34.213ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25112024-08-29T15:20:34.268ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109663 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55121 req_id = 2ae10042-1b70-4afe-a41f-eebc6f971941 response_code = 204 uri = /switch-ports
25122024-08-29T15:20:34.268ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 136 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55121 req_id = 6f52f1b5-383b-40fb-8ce4-c1597694b99a response_code = 200 uri = /network-bootstore-config
25132024-08-29T15:20:42.310ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5992 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = aa54dceb-27f3-4b0b-9930-23572b875b01 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
25142024-08-29T15:20:42.653ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:62561
25152024-08-29T15:20:42.653ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 49 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:62561 req_id = 9014572f-53a5-4114-84bc-7aba4b8529a5 response_code = 200 uri = /baseboard
25162024-08-29T15:21:03.914ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49526
25172024-08-29T15:21:03.915ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1412 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49526 req_id = 3fc4b556-a07b-4b03-b48e-b33b3981fd73 response_code = 200 uri = /vpc-routes
25182024-08-29T15:21:03.938ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 185 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:49526 req_id = 78563c1b-e7b7-4d3e-be86-1372f9e1c5f2 response_code = 204 uri = /vpc-routes
25192024-08-29T15:21:04.010ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33076
25202024-08-29T15:21:04.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 65 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33076 req_id = b20b595e-de80-49a5-897f-b4fec706ca04 response_code = 200 uri = /vpc-routes
25212024-08-29T15:21:04.022ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 143 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33076 req_id = dd996f8e-2a3f-4a54-9735-2318e9121117 response_code = 204 uri = /vpc-routes
25222024-08-29T15:21:04.139ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51677
25232024-08-29T15:21:04.139ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 61 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:51677 req_id = 41ccc955-ca4c-434c-938d-cb85ce86ffa6 response_code = 200 uri = /vpc-routes
25242024-08-29T15:21:04.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51677 req_id = bd0a2b5d-9c04-4653-9fdb-3075eeda723f response_code = 204 uri = /vpc-routes
25252024-08-29T15:21:04.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50020
25262024-08-29T15:21:04.175ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50020 req_id = f915a380-51a6-41c3-86ee-3e97ac7b2265 response_code = 200 uri = /v2p
25272024-08-29T15:21:04.195ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60086
25282024-08-29T15:21:04.195ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25292024-08-29T15:21:04.237ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25302024-08-29T15:21:04.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 106472 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:60086 req_id = 2d25a6fc-b025-4403-af7f-1d467035f4eb response_code = 204 uri = /switch-ports
25312024-08-29T15:21:04.301ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43074
25322024-08-29T15:21:04.301ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54740
25332024-08-29T15:21:04.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45516
25342024-08-29T15:21:04.302ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 257 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54740 req_id = f8938658-902e-4cfa-a9fc-6b79c3c2da50 response_code = 200 uri = /v2p
25352024-08-29T15:21:04.302ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25362024-08-29T15:21:04.302ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 340 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60086 req_id = 825d08a7-2da8-4a22-9118-023ea6a092c3 response_code = 200 uri = /network-bootstore-config
25372024-08-29T15:21:04.309ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6732 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45516 req_id = d8b8ef74-c718-470f-a828-e2679d899242 response_code = 200 uri = /v2p
25382024-08-29T15:21:04.345ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25392024-08-29T15:21:04.403ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 101019 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43074 req_id = db908637-f14d-4c8d-87cf-e3163c575441 response_code = 204 uri = /switch-ports
25402024-08-29T15:21:04.403ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43074 req_id = c102799d-71bb-4eb4-88cd-1be6f51f684a response_code = 200 uri = /network-bootstore-config
25412024-08-29T15:21:04.514ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36823
25422024-08-29T15:21:04.514ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25432024-08-29T15:21:04.560ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25442024-08-29T15:21:04.627ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113797 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:36823 req_id = 9fba97a5-92cc-4828-a5b2-0be8119483f4 response_code = 204 uri = /switch-ports
25452024-08-29T15:21:04.628ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36823 req_id = 2219d0f3-f8fc-4c33-b33d-bece02264e94 response_code = 200 uri = /network-bootstore-config
25462024-08-29T15:21:12.311ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7133 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = a8dc28bd-d61c-4293-8914-900813375209 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
25472024-08-29T15:21:12.685ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53934
25482024-08-29T15:21:12.686ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1072 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53934 req_id = e4ce2ec8-097b-4841-9f07-181445a9a7c8 response_code = 200 uri = /baseboard
25492024-08-29T15:21:33.864ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59565
25502024-08-29T15:21:33.864ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 213 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59565 req_id = 0d13db58-273a-4c30-80fa-298980b3c124 response_code = 200 uri = /v2p
25512024-08-29T15:21:33.938ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44889
25522024-08-29T15:21:33.938ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44889 req_id = b4534ffc-cc4e-4ed9-963e-8e05a20d4dce response_code = 200 uri = /vpc-routes
25532024-08-29T15:21:33.952ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 892 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:44889 req_id = 1f59a65d-1588-46c0-9f5b-13d4b3d699e8 response_code = 204 uri = /vpc-routes
25542024-08-29T15:21:33.980ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47443
25552024-08-29T15:21:33.980ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 36 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47443 req_id = 6287997a-6d29-482d-862a-2c8ab1ffdf0b response_code = 200 uri = /vpc-routes
25562024-08-29T15:21:33.995ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 92 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47443 req_id = 1b629300-9572-403e-bed4-12267f789390 response_code = 204 uri = /vpc-routes
25572024-08-29T15:21:34.009ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:33401
25582024-08-29T15:21:34.010ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 39 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:33401 req_id = 36173a76-9e62-4eae-9f28-49d614f12c54 response_code = 200 uri = /vpc-routes
25592024-08-29T15:21:34.020ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:33401 req_id = e21f7c25-0bab-43d2-a4d0-3dc8b6d300e7 response_code = 204 uri = /vpc-routes
25602024-08-29T15:21:34.036ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35554
25612024-08-29T15:21:34.036ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25622024-08-29T15:21:34.076ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25632024-08-29T15:21:34.134ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97916 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35554 req_id = a6ac29f4-7328-424c-8ef6-b23216890b3e response_code = 204 uri = /switch-ports
25642024-08-29T15:21:34.135ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46652
25652024-08-29T15:21:34.135ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 155 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46652 req_id = d9e252c0-545b-4047-a2a4-99801f44c630 response_code = 200 uri = /v2p
25662024-08-29T15:21:34.135ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35554 req_id = 078b19cb-7fda-4de1-924b-e35f530ca5fb response_code = 200 uri = /network-bootstore-config
25672024-08-29T15:21:34.137ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38463
25682024-08-29T15:21:34.137ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 233 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38463 req_id = 035e2918-0233-441a-a647-cfe7a674c0d4 response_code = 200 uri = /v2p
25692024-08-29T15:21:34.179ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63556
25702024-08-29T15:21:34.179ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25712024-08-29T15:21:34.220ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25722024-08-29T15:21:34.280ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 100967 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:63556 req_id = 1f3780cc-6cad-4843-b37b-3a210e59aa7e response_code = 204 uri = /switch-ports
25732024-08-29T15:21:34.280ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:55542
25742024-08-29T15:21:34.280ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25752024-08-29T15:21:34.315ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25762024-08-29T15:21:34.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88388 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:55542 req_id = 5a8ccc4d-f8e4-4826-b7cb-f4e43f30ac6a response_code = 204 uri = /switch-ports
25772024-08-29T15:21:34.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63556 req_id = d2a14630-6157-4160-91e0-98ee2922b75f response_code = 200 uri = /network-bootstore-config
25782024-08-29T15:21:34.369ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 88 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:55542 req_id = d23c0a3b-7b61-4461-903c-47e5de285c05 response_code = 200 uri = /network-bootstore-config
25792024-08-29T15:21:42.311ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6248 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 6647322e-d05d-4299-a198-79111773660c response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
25802024-08-29T15:21:42.719ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:56815
25812024-08-29T15:21:42.719ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 54 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:56815 req_id = 3d048e0d-b852-4211-9710-76366f08ae8d response_code = 200 uri = /baseboard
25822024-08-29T15:22:03.960ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:33331
25832024-08-29T15:22:03.961ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1248 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:33331 req_id = 143b0364-6883-4c09-9c74-7c4a0440a31e response_code = 200 uri = /vpc-routes
25842024-08-29T15:22:03.976ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 132 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:33331 req_id = af9b31e3-e2ed-411c-a761-55b9ef790d53 response_code = 204 uri = /vpc-routes
25852024-08-29T15:22:03.998ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53378
25862024-08-29T15:22:03.998ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 46 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53378 req_id = 92afc4e8-caf9-4971-a702-7e4e89e0f334 response_code = 200 uri = /vpc-routes
25872024-08-29T15:22:03.999ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56636
25882024-08-29T15:22:03.999ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 48 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56636 req_id = 84ec219c-1a76-4151-9f14-884acbeef2ca response_code = 200 uri = /vpc-routes
25892024-08-29T15:22:04.021ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1514 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:56636 req_id = d78ab234-dc86-424e-84b0-c4a91f769bfe response_code = 204 uri = /vpc-routes
25902024-08-29T15:22:04.064ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41729
25912024-08-29T15:22:04.065ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 149 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41729 req_id = 57c0b321-7d6a-4823-a19d-15f1d5d71590 response_code = 200 uri = /v2p
25922024-08-29T15:22:04.075ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:53378 req_id = d9f88dee-19c4-4997-b607-b142203c55c9 response_code = 204 uri = /vpc-routes
25932024-08-29T15:22:04.106ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64336
25942024-08-29T15:22:04.106ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64336 req_id = 634b7803-2b74-421b-9d05-7f6629ee1588 response_code = 200 uri = /v2p
25952024-08-29T15:22:04.155ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:43379
25962024-08-29T15:22:04.155ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
25972024-08-29T15:22:04.196ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
25982024-08-29T15:22:04.254ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99265 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:43379 req_id = fb261689-e2fe-4854-b943-59530ae519eb response_code = 204 uri = /switch-ports
25992024-08-29T15:22:04.254ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59617
26002024-08-29T15:22:04.254ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26012024-08-29T15:22:04.259ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4288 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:43379 req_id = ef1ef03b-e405-4251-affb-c2d39c8875a4 response_code = 200 uri = /network-bootstore-config
26022024-08-29T15:22:04.285ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:43916
26032024-08-29T15:22:04.286ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 269 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:43916 req_id = 058d6ebe-6257-4b5d-832f-d17922c942c4 response_code = 200 uri = /v2p
26042024-08-29T15:22:04.294ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26052024-08-29T15:22:04.333ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48125
26062024-08-29T15:22:04.351ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26072024-08-29T15:22:04.351ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 96852 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59617 req_id = 7121a51e-830e-4cb7-9ab6-589b062fb643 response_code = 204 uri = /switch-ports
26082024-08-29T15:22:04.357ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 690 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59617 req_id = d5d11273-1256-4fe2-91de-62e5b78e1560 response_code = 200 uri = /network-bootstore-config
26092024-08-29T15:22:04.389ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26102024-08-29T15:22:04.443ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109981 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:48125 req_id = e2921db5-e5d4-454a-bc0b-b404b48b5745 response_code = 204 uri = /switch-ports
26112024-08-29T15:22:04.444ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48125 req_id = 98f47fc8-e781-43da-a93f-2c6bb6470ebb response_code = 200 uri = /network-bootstore-config
26122024-08-29T15:22:12.321ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6626 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 290dd813-2e0c-45b7-8ccf-4945ccd49958 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
26132024-08-29T15:22:12.754ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:44985
26142024-08-29T15:22:12.754ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 546 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:44985 req_id = baa32853-fbeb-4692-91f6-22488b508187 response_code = 200 uri = /baseboard
26152024-08-29T15:22:33.825ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47755
26162024-08-29T15:22:33.825ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47755 req_id = 69be1e7a-7439-412e-9d19-9f544dbe8e26 response_code = 200 uri = /vpc-routes
26172024-08-29T15:22:33.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 178 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:47755 req_id = c21aea3e-44bc-48d6-856a-e5ff13e46ee6 response_code = 204 uri = /vpc-routes
26182024-08-29T15:22:33.839ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61269
26192024-08-29T15:22:33.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61269 req_id = 809b8724-9ce6-49a1-b632-471c83638990 response_code = 200 uri = /vpc-routes
26202024-08-29T15:22:33.849ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1164 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61269 req_id = 029e4e34-a7d7-4178-a70b-661f2b1d7d2b response_code = 204 uri = /vpc-routes
26212024-08-29T15:22:33.895ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42365
26222024-08-29T15:22:33.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 73 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42365 req_id = 4226f0b8-33be-4902-9b4d-9928453925db response_code = 200 uri = /vpc-routes
26232024-08-29T15:22:33.906ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 154 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:42365 req_id = b0c38feb-f9fa-4dd6-8c98-7a1550f999ba response_code = 204 uri = /vpc-routes
26242024-08-29T15:22:33.943ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:41187
26252024-08-29T15:22:33.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 133 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:41187 req_id = e79b696f-b01d-4f41-9a10-155e56fdf44c response_code = 200 uri = /v2p
26262024-08-29T15:22:33.972ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59908
26272024-08-29T15:22:33.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59908 req_id = 56f26639-caea-4542-b3ae-9a3cf8966a21 response_code = 200 uri = /v2p
26282024-08-29T15:22:33.974ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34170
26292024-08-29T15:22:33.974ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26302024-08-29T15:22:34.015ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26312024-08-29T15:22:34.083ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109402 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34170 req_id = 19027d6a-245e-45c2-9b2f-9e1c3cf624a8 response_code = 204 uri = /switch-ports
26322024-08-29T15:22:34.084ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:46730
26332024-08-29T15:22:34.084ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43441
26342024-08-29T15:22:34.084ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 351 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34170 req_id = d8882da7-5c02-453b-95d0-cfe01c72a1cc response_code = 200 uri = /network-bootstore-config
26352024-08-29T15:22:34.085ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26362024-08-29T15:22:34.089ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40069
26372024-08-29T15:22:34.089ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 234 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40069 req_id = 46282195-6269-41ad-9d2d-a7bafd24ff5e response_code = 200 uri = /v2p
26382024-08-29T15:22:34.123ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26392024-08-29T15:22:34.179ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26402024-08-29T15:22:34.179ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 95355 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:46730 req_id = 1b331eeb-b4ab-4c62-bf13-680942e317ed response_code = 204 uri = /switch-ports
26412024-08-29T15:22:34.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 310 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:46730 req_id = 8793814e-adb6-47d2-b806-76b9b4ac7780 response_code = 200 uri = /network-bootstore-config
26422024-08-29T15:22:34.214ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26432024-08-29T15:22:34.271ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 187496 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43441 req_id = de8f7fe7-8b5e-451e-b463-483eab7412b8 response_code = 204 uri = /switch-ports
26442024-08-29T15:22:34.272ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 686 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43441 req_id = 9c5863bb-d1d5-4ae8-b402-b3f951051246 response_code = 200 uri = /network-bootstore-config
26452024-08-29T15:22:42.391ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7212 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 3bedaa53-4c87-4ad8-a72e-c1c73c989bba response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
26462024-08-29T15:22:42.787ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59699
26472024-08-29T15:22:42.787ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59699 req_id = 86f82d6f-e9fb-4d28-8d13-8e5d5190b50e response_code = 200 uri = /baseboard
26482024-08-29T15:23:03.904ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53409
26492024-08-29T15:23:03.904ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53409 req_id = c83103be-6029-4196-b20a-cb3e23a9c407 response_code = 200 uri = /vpc-routes
26502024-08-29T15:23:03.916ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:53409 req_id = bfa54c6d-16dc-477f-be40-7dbefba5c4b0 response_code = 204 uri = /vpc-routes
26512024-08-29T15:23:03.934ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:51222
26522024-08-29T15:23:03.935ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1210 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:51222 req_id = 33d3b0a9-9ce8-4fa1-93f7-594f3e500c3c response_code = 200 uri = /vpc-routes
26532024-08-29T15:23:03.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56325
26542024-08-29T15:23:03.990ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 161 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56325 req_id = a8fd07b0-a95c-4422-a922-2fe1a8b62729 response_code = 200 uri = /v2p
26552024-08-29T15:23:03.997ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:51222 req_id = 731b71a6-1465-4430-8875-4575bda175ca response_code = 204 uri = /vpc-routes
26562024-08-29T15:23:04.044ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48066
26572024-08-29T15:23:04.044ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26582024-08-29T15:23:04.091ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26592024-08-29T15:23:04.155ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 111188 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48066 req_id = 034efbb7-6e1f-4ad9-8007-eea8116ed37e response_code = 204 uri = /switch-ports
26602024-08-29T15:23:04.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41243
26612024-08-29T15:23:04.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:56541
26622024-08-29T15:23:04.156ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53254
26632024-08-29T15:23:04.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 169 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48066 req_id = 22fa1e5c-f782-4adf-a4fb-0a0839255993 response_code = 200 uri = /network-bootstore-config
26642024-08-29T15:23:04.156ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26652024-08-29T15:23:04.156ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 316 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:56541 req_id = 8e2e4f9d-70c5-44ce-ac40-5e786e3a01f5 response_code = 200 uri = /v2p
26662024-08-29T15:23:04.160ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4481 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53254 req_id = 619db22e-2cec-4898-ad07-370481373766 response_code = 200 uri = /vpc-routes
26672024-08-29T15:23:04.176ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:40893
26682024-08-29T15:23:04.176ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 141 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:40893 req_id = 70f5ce12-de7d-4532-86f1-03cd69b2c426 response_code = 200 uri = /v2p
26692024-08-29T15:23:04.179ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53254 req_id = 227b1076-ac59-4544-aec3-221eb8c5dfc5 response_code = 204 uri = /vpc-routes
26702024-08-29T15:23:04.191ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26712024-08-29T15:23:04.218ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54022
26722024-08-29T15:23:04.249ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26732024-08-29T15:23:04.249ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 93372 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41243 req_id = 17691271-02fb-4d60-9397-6a8a08e5a047 response_code = 204 uri = /switch-ports
26742024-08-29T15:23:04.253ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 578 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41243 req_id = cd6edf79-a71d-44f8-8c0c-aa683b5f8cdf response_code = 200 uri = /network-bootstore-config
26752024-08-29T15:23:04.284ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26762024-08-29T15:23:04.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 123611 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:54022 req_id = 37de9358-d37f-44ed-9eec-3d95c9fa7b69 response_code = 204 uri = /switch-ports
26772024-08-29T15:23:04.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54022 req_id = bde9c3c6-f65c-446c-83c9-fbd45e133e01 response_code = 200 uri = /network-bootstore-config
26782024-08-29T15:23:12.390ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5666 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 5f5467dc-a92c-4e8f-8cad-757bd60c8ba4 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
26792024-08-29T15:23:12.820ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45021
26802024-08-29T15:23:12.820ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 120 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45021 req_id = 0f2def4e-8723-490e-9b35-8b1605186870 response_code = 200 uri = /baseboard
26812024-08-29T15:23:33.731ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61893
26822024-08-29T15:23:33.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 129 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61893 req_id = 4b7e4dbf-8f6d-4500-9c48-a2ec9b0d703b response_code = 200 uri = /vpc-routes
26832024-08-29T15:23:33.746ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61893 req_id = f585760a-00c8-40f1-acd2-8325a0b61ec5 response_code = 204 uri = /vpc-routes
26842024-08-29T15:23:33.814ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44795
26852024-08-29T15:23:33.814ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44795 req_id = 9624e2b5-2770-4ed5-a5b5-84c502dbaa74 response_code = 200 uri = /vpc-routes
26862024-08-29T15:23:33.826ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:44795 req_id = 82c7c72f-d8a9-4284-944a-f9d5e8c30594 response_code = 204 uri = /vpc-routes
26872024-08-29T15:23:33.872ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63545
26882024-08-29T15:23:33.872ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63545 req_id = faae3eca-4c66-435a-8444-e31a3acb5c3d response_code = 200 uri = /vpc-routes
26892024-08-29T15:23:33.882ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:63545 req_id = 6c6f2a84-d5bb-4a3d-b823-696c9bdf53d5 response_code = 204 uri = /vpc-routes
26902024-08-29T15:23:33.909ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37905
26912024-08-29T15:23:33.909ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
26922024-08-29T15:23:33.948ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
26932024-08-29T15:23:34.011ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 102251 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37905 req_id = 82ed595f-6a34-42b7-b3f5-218fb16933bc response_code = 204 uri = /switch-ports
26942024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41655
26952024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34559
26962024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50823
26972024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45770
26982024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34912
26992024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 270 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37905 req_id = a7325f48-2e25-4215-a7d4-6509f9861275 response_code = 200 uri = /network-bootstore-config
27002024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 254 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45770 req_id = 5f11d982-bda6-4402-999d-738fd2d9310b response_code = 200 uri = /v2p
27012024-08-29T15:23:34.012ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 339 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34559 req_id = dc496e64-be31-46ad-8860-28e7319c481c response_code = 200 uri = /v2p
27022024-08-29T15:23:34.013ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27032024-08-29T15:23:34.013ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1214 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50823 req_id = 51acc129-17bf-4f05-a6a2-7576a0984951 response_code = 200 uri = /v2p
27042024-08-29T15:23:34.054ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27052024-08-29T15:23:34.111ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27062024-08-29T15:23:34.111ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 98732 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34912 req_id = 0df91bad-6909-425a-8a27-b536da6e0d00 response_code = 204 uri = /switch-ports
27072024-08-29T15:23:34.115ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 365 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34912 req_id = 0b3fc78c-7f17-4b00-a488-728ee58c44b4 response_code = 200 uri = /network-bootstore-config
27082024-08-29T15:23:34.146ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27092024-08-29T15:23:34.206ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 194457 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:41655 req_id = bf458bb5-b117-406c-974d-8dcfbc6148e5 response_code = 204 uri = /switch-ports
27102024-08-29T15:23:34.207ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 175 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41655 req_id = 2cff1fa1-07bd-4d48-a4e8-253c3ac7aa1f response_code = 200 uri = /network-bootstore-config
27112024-08-29T15:23:42.394ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 7715 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 0971aced-3e75-44c4-92fb-5453749bb53d response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
27122024-08-29T15:23:42.851ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48068
27132024-08-29T15:23:42.851ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 42 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48068 req_id = 259e1a89-7394-4c1a-9533-1476c8bfa253 response_code = 200 uri = /baseboard
27142024-08-29T15:24:03.880ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38104
27152024-08-29T15:24:03.880ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38104 req_id = 6f96efe9-6f62-4932-b280-ac826455ce0c response_code = 200 uri = /vpc-routes
27162024-08-29T15:24:03.896ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1890 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38104 req_id = ec1e1b4b-83ed-4e97-bc49-00acb92c41aa response_code = 204 uri = /vpc-routes
27172024-08-29T15:24:04.011ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54873
27182024-08-29T15:24:04.011ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 67 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:54873 req_id = 3215403a-18f1-4cc7-a2e2-96887b761f93 response_code = 200 uri = /vpc-routes
27192024-08-29T15:24:04.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60783
27202024-08-29T15:24:04.102ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 83 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60783 req_id = 86e722df-8067-4dbc-b1db-68392545cc04 response_code = 200 uri = /vpc-routes
27212024-08-29T15:24:04.117ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 137 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:60783 req_id = 6290e15c-b52b-4c38-bf53-8c28c0dbd3b9 response_code = 204 uri = /vpc-routes
27222024-08-29T15:24:04.146ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42113
27232024-08-29T15:24:04.146ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 193 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42113 req_id = 8f1b7196-c3fe-41d5-a7e5-c46b4f832c15 response_code = 200 uri = /v2p
27242024-08-29T15:24:04.175ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47598
27252024-08-29T15:24:04.175ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27262024-08-29T15:24:04.231ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27272024-08-29T15:24:04.301ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 126351 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47598 req_id = 6935ec7a-30d2-43d1-8753-1a81cb9941d5 response_code = 204 uri = /switch-ports
27282024-08-29T15:24:04.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55757
27292024-08-29T15:24:04.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:59998
27302024-08-29T15:24:04.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56873
27312024-08-29T15:24:04.302ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34494
27322024-08-29T15:24:04.302ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1187 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54873 req_id = aaab9fee-2c49-44ad-95e1-0af2f0812abb response_code = 204 uri = /vpc-routes
27332024-08-29T15:24:04.303ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27342024-08-29T15:24:04.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 958 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34494 req_id = d30d607a-8135-4e88-88cb-dcd7527b4597 response_code = 200 uri = /v2p
27352024-08-29T15:24:04.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1225 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47598 req_id = 4363a001-04cb-4226-9cc3-df4f2d7307b2 response_code = 200 uri = /network-bootstore-config
27362024-08-29T15:24:04.303ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 982 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56873 req_id = 4454a32c-fd90-4714-be7d-3c1c33ddb942 response_code = 200 uri = /v2p
27372024-08-29T15:24:04.303ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27382024-08-29T15:24:04.303ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27392024-08-29T15:24:04.308ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27402024-08-29T15:24:04.308ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27412024-08-29T15:24:04.308ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27422024-08-29T15:24:04.308ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
27432024-08-29T15:24:04.308ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27442024-08-29T15:24:04.309ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6826 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55757 req_id = 9f206adb-c69c-48b3-8fc2-0c984cd29a02 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27452024-08-29T15:24:04.310ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58935
27462024-08-29T15:24:04.346ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27472024-08-29T15:24:04.358ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34366
27482024-08-29T15:24:04.362ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27492024-08-29T15:24:04.363ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27502024-08-29T15:24:04.363ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27512024-08-29T15:24:04.363ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27522024-08-29T15:24:04.363ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27532024-08-29T15:24:04.363ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
27542024-08-29T15:24:04.363ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27552024-08-29T15:24:04.363ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 4846 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:34366 req_id = 93ad66b1-b2ea-438c-90ad-2f7b95be44ba response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27562024-08-29T15:24:04.407ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27572024-08-29T15:24:04.407ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 105559 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:59998 req_id = 66a783d8-e61a-4c5a-a0f8-fe68c0cfb63e response_code = 204 uri = /switch-ports
27582024-08-29T15:24:04.413ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 769 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:59998 req_id = 7479bbeb-19f2-419d-91a1-26037ffba882 response_code = 200 uri = /network-bootstore-config
27592024-08-29T15:24:04.448ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27602024-08-29T15:24:04.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 197279 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58935 req_id = b040a7c6-5ffc-47c7-a114-f8feca9ff916 response_code = 204 uri = /switch-ports
27612024-08-29T15:24:04.508ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58935 req_id = 9c29c873-fe9b-4f37-837c-98356f54c90e response_code = 200 uri = /network-bootstore-config
27622024-08-29T15:24:04.552ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:61647
27632024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:671 rules = [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 9a1c332a-d0f3-4bf0-a446-7708135ac57c, kind: Service { id: 49d104a1-1335-4930-9b9e-1ec4a3ca16f9 }, name: Name("external-dns-49d104a1-1335-4930-9b9e-1ec4a3ca16f9"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 188, 52])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: ea86e1a2-b347-400f-a58e-d0e965a26ce3, kind: Service { id: 45f571be-936e-48ca-b02e-6bbc2b2f1974 }, name: Name("external-dns-45f571be-936e-48ca-b02e-6bbc2b2f1974"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 217, 255])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 58c95a6c-0b6d-4e8e-9ca9-eb44bbbbd1be, kind: Service { id: a66efbaf-5f9a-40ef-9401-e26ed785dbe9 }, name: Name("nexus-a66efbaf-5f9a-40ef-9401-e26ed785dbe9"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 134, 65])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 65a401da-ec12-42f3-877e-1b2d9a2cf041, kind: Service { id: d36d5399-1e8d-42d4-b1c6-82ab39a196e5 }, name: Name("nexus-d36d5399-1e8d-42d4-b1c6-82ab39a196e5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 132, 224])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }, NetworkInterface { id: 8f376814-87ad-4ea2-9c55-936c27a65b97, kind: Service { id: a752201f-8e19-45ac-9901-65317e43fc2e }, name: Name("nexus-a752201f-8e19-45ac-9901-65317e43fc2e"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 235, 245])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
27642024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27652024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27662024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
27672024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27682024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte0" rules = []
27692024-08-29T15:24:04.552ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:690 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
27702024-08-29T15:24:04.552ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 233 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:61647 req_id = 7bac4ed7-5094-4a5f-a0f1-fb6d2a7d9de7 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27712024-08-29T15:24:12.390ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6313 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 6c918bb0-6e25-48fe-bf68-dffec1753f21 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
27722024-08-29T15:24:12.886ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33341
27732024-08-29T15:24:12.886ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 47 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33341 req_id = 4fc4ad8a-66a7-4647-9da4-dd357c67b340 response_code = 200 uri = /baseboard
27742024-08-29T15:24:15.103ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:197
27752024-08-29T15:24:15.103ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
27762024-08-29T15:24:15.146ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:186 next_cleanup = Instant { tv_sec: 21373, tv_nsec: 890481498 } time_to_next_cleanup = 599.99999578s
27772024-08-29T15:24:29.435ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:862
27782024-08-29T15:24:33.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34038
27792024-08-29T15:24:33.720ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 1622 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34038 req_id = 34fc5c7f-602a-43ec-96a3-6edb39603896 response_code = 200 uri = /vpc-routes
27802024-08-29T15:24:33.740ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 161 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34038 req_id = cba8f061-5c7f-4d25-be7c-69f58b879ecd response_code = 204 uri = /vpc-routes
27812024-08-29T15:24:33.822ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:63524
27822024-08-29T15:24:33.824ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 55 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:63524 req_id = 047ab071-4749-4a4d-bcc6-fa676ba6bce1 response_code = 200 uri = /vpc-routes
27832024-08-29T15:24:33.838ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:63524 req_id = dc87ad06-a91c-4a84-92a9-b6da441cac18 response_code = 204 uri = /vpc-routes
27842024-08-29T15:24:33.899ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62760
27852024-08-29T15:24:33.899ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 165 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62760 req_id = 99ef14e6-ad05-4fe9-9749-12a4eab8b8f3 response_code = 200 uri = /v2p
27862024-08-29T15:24:33.922ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44026
27872024-08-29T15:24:33.922ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27882024-08-29T15:24:33.964ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27892024-08-29T15:24:34.032ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 110864 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:44026 req_id = 7d8063a4-3de0-4ef4-b56f-2fe4d4d9ab2e response_code = 204 uri = /switch-ports
27902024-08-29T15:24:34.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33333
27912024-08-29T15:24:34.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56174
27922024-08-29T15:24:34.033ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35526
27932024-08-29T15:24:34.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33333 req_id = 57b81636-b06f-448d-9942-ddb215dfd709 response_code = 200 uri = /vpc-routes
27942024-08-29T15:24:34.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 264 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44026 req_id = fc90ecaa-25e9-4eba-b68c-22d27ec46f63 response_code = 200 uri = /network-bootstore-config
27952024-08-29T15:24:34.033ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 325 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56174 req_id = 9a5fc231-7e02-4431-8017-b1ac00938c63 response_code = 200 uri = /v2p
27962024-08-29T15:24:34.034ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
27972024-08-29T15:24:34.043ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 307 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:33333 req_id = d2a96b78-d619-435c-b1f5-f1813f344a59 response_code = 204 uri = /vpc-routes
27982024-08-29T15:24:34.080ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
27992024-08-29T15:24:34.098ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56504
28002024-08-29T15:24:34.098ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 273 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56504 req_id = 0d9f7765-f39c-4888-a447-6ffa200af4a8 response_code = 200 uri = /v2p
28012024-08-29T15:24:34.114ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57369
28022024-08-29T15:24:34.145ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
28032024-08-29T15:24:34.145ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 112307 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35526 req_id = 194effb3-a49e-4fee-afe7-30bd701ebccd response_code = 204 uri = /switch-ports
28042024-08-29T15:24:34.150ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 657 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35526 req_id = bbd1b2b8-82f3-4e82-9166-823321a3fbed response_code = 200 uri = /network-bootstore-config
28052024-08-29T15:24:34.185ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
28062024-08-29T15:24:34.246ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 131936 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57369 req_id = 3e3775a5-b9a2-4889-a32b-278bedf20fe1 response_code = 204 uri = /switch-ports
28072024-08-29T15:24:34.246ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57369 req_id = 2e4924e7-15db-4d84-aa6b-c81c2d4d6146 response_code = 200 uri = /network-bootstore-config
28082024-08-29T15:24:42.390ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6063 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = bada2397-aff2-4c7a-bd02-958f17fcdda8 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
28092024-08-29T15:24:42.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34612
28102024-08-29T15:24:42.609ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 128 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34612 req_id = f3250ae0-cea3-4d86-a0a0-72883794d780 response_code = 200 uri = /v2p
28112024-08-29T15:24:42.610ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:778 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(7604531) }
28122024-08-29T15:24:42.610ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 630 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:34612 req_id = 54d34ca8-0b93-4ed6-9b5e-9894ebf12682 response_code = 204 uri = /v2p
28132024-08-29T15:24:42.718ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62095
28142024-08-29T15:24:42.720ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:570 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(7604531))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-5788b950fad", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 2624347c-7537-49af-8ab7-cdee418cb0cc, read_only_parent: Some(Volume { block_size: 512, id: 3834d461-5285-4dc4-8d41-e0ac17dcc16f, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 387724df-00cf-41bf-93e8-43ee618611a9, key: Some("4ZJaAAbgLboYhE+wnCeyBEZfLToOAYoqqBAr0xTDJ68="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 2624347c-7537-49af-8ab7-cdee418cb0cc, key: Some("wJiRsXlDbVgqH0sx31rnOruxThpcUI/txHGg4G/M5no="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19000"] } }] } }], cloud_init_bytes: Some(..) } instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da instance_runtime = InstanceRuntimeState { propolis_id: Some(88d1c68d-bdb0-4b5b-82b9-a01a804ae13c (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-08-29T15:24:42.216581335Z } metadata = InstanceMetadata { silo_id: 50153162-da63-464a-9827-7da368e797d8, project_id: 8a74de03-5df8-4ae4-87eb-7eb972345fae } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-29T15:24:42.140203Z }
28152024-08-29T15:24:42.720ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:603 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28162024-08-29T15:24:42.720ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:991 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da migration_id = None propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], filter_hosts: Some([Vpc(Vni(7604531))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, ResolvedVpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 574b66c1-2c73-470c-8c17-6aeeff1266b9, kind: Instance { id: dea8d9be-c41c-461e-a0e0-e5b03267d5da }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 245, 121, 135])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(7604531), primary: true, slot: 0, transit_ips: [] }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-5788b950fad", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 2624347c-7537-49af-8ab7-cdee418cb0cc, read_only_parent: Some(Volume { block_size: 512, id: 3834d461-5285-4dc4-8d41-e0ac17dcc16f, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 387724df-00cf-41bf-93e8-43ee618611a9, key: Some("4ZJaAAbgLboYhE+wnCeyBEZfLToOAYoqqBAr0xTDJ68="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::16]:19001", "[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::15]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 2624347c-7537-49af-8ab7-cdee418cb0cc, key: Some("wJiRsXlDbVgqH0sx31rnOruxThpcUI/txHGg4G/M5no="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::1a]:19001", "[fd00:1122:3344:101::19]:19001", "[fd00:1122:3344:101::17]:19000"] } }] } }], cloud_init_bytes: Some(..) }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-08-29T15:24:42.140203Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400, migration_id: None }
28172024-08-29T15:24:42.721ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 2725 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62095 req_id = 2ca21154-66cb-4521-96ba-e765b87a7089 response_code = 200 uri = /vmms/88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28182024-08-29T15:24:42.795ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62824
28192024-08-29T15:24:42.795ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62824 req_id = 2fbbeede-1006-475b-a72e-107a6494c553 response_code = 200 uri = /vmms/88d1c68d-bdb0-4b5b-82b9-a01a804ae13c/state
28202024-08-29T15:24:42.796ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:424 port = Port { inner: PortInner(PortData { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 245, 121, 135]), slot: 0, vni: Vni { inner: 7604531 }, subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 } }) }
28212024-08-29T15:24:42.796ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
28222024-08-29T15:24:42.834ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c file = illumos-utils/src/zone.rs:316 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28232024-08-29T15:24:42.858ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c file = illumos-utils/src/zone.rs:354 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28242024-08-29T15:24:42.918ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51583
28252024-08-29T15:24:42.918ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 107 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51583 req_id = 64d4e5fa-8040-4ae6-b919-5be6d50cdea7 response_code = 200 uri = /baseboard
28262024-08-29T15:24:43.841ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value='oxControlInstance0'/> <propval type="astring" name="gateway" value='fd00:1122:3344:101::1'/> <propval type="astring" name="listen_addr" value='fd00:1122:3344:101::1:0'/> <propval type="astring" name="listen_port" value='12400'/> <propval type="astring" name="metric_addr" value='dns'/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28272024-08-29T15:24:43.841ZINFOSledAgent (InstanceManager): Booting oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone file = illumos-utils/src/running_zone.rs:488 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28282024-08-29T15:24:44.058ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 60.662009ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28292024-08-29T15:24:44.143ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 94.693895ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28302024-08-29T15:24:44.254ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 164.720714ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28312024-08-29T15:24:44.435ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 537.568705ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28322024-08-29T15:24:44.989ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 506.380155ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28332024-08-29T15:24:45.508ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 1.230023474s file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28342024-08-29T15:24:46.750ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 1.478103269s file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28352024-08-29T15:24:48.239ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 1.012909382s file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c zone = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28362024-08-29T15:24:49.280ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c file = sled-agent/src/instance.rs:1449 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28372024-08-29T15:24:49.288ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 61.498647ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28382024-08-29T15:24:49.359ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 132.282119ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28392024-08-29T15:24:49.501ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c") failed: Property not found. retry in 235.059755ms file = illumos-utils/src/svc.rs:36 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28402024-08-29T15:24:49.749ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1458 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28412024-08-29T15:24:49.787ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1482 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28422024-08-29T15:24:49.933ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:727 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28432024-08-29T15:24:49.936ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Creating), migration_in: None, migration_out: None, time: 2024-08-29T15:24:49.936481141Z } file = sled-agent/src/instance.rs:594 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28442024-08-29T15:24:49.936ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-29T15:24:49.936481141Z } propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28452024-08-29T15:24:49.936ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c state = SledVmmState { vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-08-29T15:24:49.936481141Z }, migration_in: None, migration_out: None }
28462024-08-29T15:24:49.954ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_in: None, migration_out: None, time: 2024-08-29T15:24:49.954060186Z } file = sled-agent/src/instance.rs:594 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28472024-08-29T15:24:49.954ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-29T15:24:49.954060186Z } propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28482024-08-29T15:24:49.954ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c state = SledVmmState { vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-08-29T15:24:49.954060186Z }, migration_in: None, migration_out: None }
28492024-08-29T15:25:03.907ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54979
28502024-08-29T15:25:03.907ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 223 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54979 req_id = 163aa6f7-ba27-4250-b4da-a067ab15739b response_code = 200 uri = /vmms/88d1c68d-bdb0-4b5b-82b9-a01a804ae13c/state
28512024-08-29T15:25:04.020ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56498
28522024-08-29T15:25:04.020ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 144 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56498 req_id = 539ebcb0-d5e2-4ef3-804f-eccde164dc36 response_code = 200 uri = /vmms/88d1c68d-bdb0-4b5b-82b9-a01a804ae13c/state
28532024-08-29T15:25:04.039ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61869
28542024-08-29T15:25:04.040ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 66 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61869 req_id = 4fdea449-ae7e-4b4a-b6a4-5b990aea894a response_code = 200 uri = /vpc-routes
28552024-08-29T15:25:04.074ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:50634
28562024-08-29T15:25:04.075ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 664 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:50634 req_id = 98634cd8-c191-4702-8fae-290e16b68085 response_code = 200 uri = /vmms/88d1c68d-bdb0-4b5b-82b9-a01a804ae13c/state
28572024-08-29T15:25:04.095ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37411
28582024-08-29T15:25:04.095ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 44 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37411 req_id = c3b07144-0e9d-4d89-b6dd-90189aaba97d response_code = 200 uri = /vpc-routes
28592024-08-29T15:25:04.125ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:51293
28602024-08-29T15:25:04.126ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:51293 req_id = 162a6af8-8c53-4871-bdfd-e31881093246 response_code = 200 uri = /vpc-routes
28612024-08-29T15:25:04.179ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61572
28622024-08-29T15:25:04.179ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 186 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61572 req_id = 1360f907-3938-4ba1-b8f3-dec89c380fc9 response_code = 200 uri = /v2p
28632024-08-29T15:25:04.183ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:44916
28642024-08-29T15:25:04.183ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:44916 req_id = b0a20429-9bc9-4462-bf6c-42c7f0cd3509 response_code = 200 uri = /v2p
28652024-08-29T15:25:04.230ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 263 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:37411 req_id = 1577d1b6-0cd1-41e1-a2a2-db9ed5cd26ba response_code = 204 uri = /vpc-routes
28662024-08-29T15:25:04.232ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53823
28672024-08-29T15:25:04.233ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
28682024-08-29T15:25:04.274ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
28692024-08-29T15:25:04.332ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 99660 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53823 req_id = 7a77288f-8cdb-48dc-b611-e03f55ab99b0 response_code = 204 uri = /switch-ports
28702024-08-29T15:25:04.332ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47357
28712024-08-29T15:25:04.332ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:37968
28722024-08-29T15:25:04.332ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 318 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:51293 req_id = 6d55019e-f674-4ff6-a546-91ed82fc8811 response_code = 204 uri = /vpc-routes
28732024-08-29T15:25:04.332ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:54960
28742024-08-29T15:25:04.338ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5289 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:61869 req_id = 4c81dc13-79e9-4332-9ca4-22f8da55e292 response_code = 204 uri = /vpc-routes
28752024-08-29T15:25:04.338ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 5512 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53823 req_id = dbee400a-b9f5-4472-b087-a70f21b07434 response_code = 200 uri = /network-bootstore-config
28762024-08-29T15:25:04.338ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
28772024-08-29T15:25:04.338ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 502 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:54960 req_id = 7445e8d0-c193-4051-bcce-53f9a462f74f response_code = 200 uri = /v2p
28782024-08-29T15:25:04.377ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
28792024-08-29T15:25:04.435ZINFOSledAgent (ServiceManager): ensuring scrimlet uplinks file = sled-agent/src/services.rs:4038
28802024-08-29T15:25:04.435ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 97015 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47357 req_id = 203f3f29-fec0-43fe-aef4-2a4df47d39a0 response_code = 204 uri = /switch-ports
28812024-08-29T15:25:04.441ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 616 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47357 req_id = 6e5867df-cdcf-419b-abae-b96d92306569 response_code = 200 uri = /network-bootstore-config
28822024-08-29T15:25:04.473ZINFOSledAgent (ServiceManager): setting up port_qsfp0 file = sled-agent/src/services.rs:4062
28832024-08-29T15:25:04.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 190790 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:37968 req_id = d0758f5c-bf04-46ba-9026-3f62dc6d0d3d response_code = 204 uri = /switch-ports
28842024-08-29T15:25:04.529ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 162 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:37968 req_id = 30f82f78-e5fe-4e6c-86c3-21966c485fad response_code = 200 uri = /network-bootstore-config
28852024-08-29T15:25:07.838ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52350
28862024-08-29T15:25:07.839ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 704 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52350 req_id = b02315f6-a381-4037-a5ab-c29dc62a4a34 response_code = 200 uri = /vmms/88d1c68d-bdb0-4b5b-82b9-a01a804ae13c/state
28872024-08-29T15:25:07.839ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_in: None, migration_out: None, time: 2024-08-29T15:25:07.839582579Z } file = sled-agent/src/instance.rs:594 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28882024-08-29T15:25:07.839ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-29T15:25:07.839582579Z } propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28892024-08-29T15:25:07.839ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c state = SledVmmState { vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-08-29T15:25:07.839582579Z }, migration_in: None, migration_out: None }
28902024-08-29T15:25:07.879ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_in: None, migration_out: None, time: 2024-08-29T15:25:07.879362368Z } file = sled-agent/src/instance.rs:594 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28912024-08-29T15:25:07.879ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:616 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-08-29T15:25:07.879362368Z } propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28922024-08-29T15:25:07.879ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:630 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28932024-08-29T15:25:07.879ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
28942024-08-29T15:25:07.880ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 5755691835433126538 n_samples = 6 sled_id = b9c40332-a231-42db-adf0-b267d663a311
28952024-08-29T15:25:07.880ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:451
28962024-08-29T15:25:07.880ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/5a85a3ef-48f8-4333-ae7e-46dfaa7fa523/debug/bundle/zone", "/pool/int/6dd67a4e-189f-47cd-a6e2-0eff97293dab/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/05effe78-0c08-40b4-916b-cde344744ec3/crypt/debug", "/pool/ext/866d818f-82bb-444c-992a-1351dec91efa/crypt/debug", "/pool/ext/4fbbfff0-e175-4404-8cd9-acfc358b7140/crypt/debug", "/pool/ext/6eaf0007-0aa9-44b6-8bac-b01569f4c6a3/crypt/debug", "/pool/ext/775439b8-d6ea-4b72-9708-a30c4c7c6d6f/crypt/debug", "/pool/ext/e4f77dae-534d-4b73-b38a-890d525d3747/crypt/debug", "/pool/ext/1b55399c-9dbb-4d32-bd51-426a7996ea24/crypt/debug", "/pool/ext/aad99d77-5c0c-4ae7-8cc1-ced403d6b98a/crypt/debug", "/pool/ext/f4912882-4c2a-4c0d-b986-5cbff960587e/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:339 zone_name = oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
28972024-08-29T15:25:07.880ZWARNSledAgent (kstat-sampler-worker): target removed with queued samples file = oximeter/instruments/src/kstat/sampler.rs:557 id = 6325694712850844852 n_samples = 6 sled_id = b9c40332-a231-42db-adf0-b267d663a311
28982024-08-29T15:25:08.507ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1137 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c", bundle_id: 30d1d231-0287-4f3d-bbfc-f4bbf08de0e7 }, time_created: 2024-08-29T15:25:07.881197054Z, version: 0, cause: TerminatedInstance }
28992024-08-29T15:25:08.507ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_88d1c68d-bdb0-4b5b-82b9-a01a804ae13c file = sled-agent/src/instance.rs:826 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
29002024-08-29T15:25:10.273ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:270 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
29012024-08-29T15:25:10.291ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(7604531), kind: System }
29022024-08-29T15:25:10.292ZINFOSledAgent (PortManager): Removed route set for subnet file = illumos-utils/src/opte/port_manager.rs:904 id = RouterId { vni: Vni(7604531), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }
29032024-08-29T15:25:10.292ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-29T15:25:10.292369135Z }, migration_in: None, migration_out: None }
29042024-08-29T15:25:10.344ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:514 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c state = SledVmmState { vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-08-29T15:25:10.292369135Z }, migration_in: None, migration_out: None }
29052024-08-29T15:25:10.349ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:763 instance_id = dea8d9be-c41c-461e-a0e0-e5b03267d5da propolis_id = 88d1c68d-bdb0-4b5b-82b9-a01a804ae13c
29062024-08-29T15:25:10.780ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:40160
29072024-08-29T15:25:10.780ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 70 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:40160 req_id = 26445dd6-3ff1-417b-949d-1967cff8ba7d response_code = 200 uri = /vpc-routes
29082024-08-29T15:25:10.789ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:40160 req_id = 426cb7b3-e7a6-457f-9e3a-71a955f6e0fe response_code = 204 uri = /vpc-routes
29092024-08-29T15:25:10.991ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58390
29102024-08-29T15:25:10.991ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 146 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58390 req_id = 04bb7609-e751-483c-aa2d-2ba84123656d response_code = 200 uri = /v2p
29112024-08-29T15:25:11.312ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65131
29122024-08-29T15:25:11.312ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65131 req_id = fbba3dad-17d4-41d0-af24-21aaecc8957f response_code = 200 uri = /v2p
29132024-08-29T15:25:12.505ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 6084 local_addr = [fd00:1122:3344:101::1]:34899 method = GET remote_addr = [fd00:1122:3344:101::d]:51670 req_id = 32d0b532-5747-41b4-8094-a759ce9bed42 response_code = 200 sled_id = b9c40332-a231-42db-adf0-b267d663a311 uri = /b9c40332-a231-42db-adf0-b267d663a311
29142024-08-29T15:25:12.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:59296
29152024-08-29T15:25:12.647ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 80 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:59296 req_id = 45e95119-4cd1-4998-bd64-7b3295a94e81 response_code = 200 uri = /vpc-routes
29162024-08-29T15:25:12.655ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:59296 req_id = afef4074-c109-4618-9b93-de298ac444c5 response_code = 204 uri = /vpc-routes
29172024-08-29T15:25:12.953ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:775 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37872
29182024-08-29T15:25:12.953ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/06c8dab/dropshot/src/server.rs:914 latency_us = 41 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37872 req_id = 8535ae4a-a52c-44ba-beb4-8df52ad1d8fe response_code = 200 uri = /baseboard