Max level shown:
1 [ Jun 21 21:37:11 Enabled. ]
2 [ Jun 21 21:37:12 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml &"). ]
3 [ Jun 21 21:37:12 Method "start" exited with status 0. ]
4 [ Jun 21 21:37:12 Rereading configuration. ]
5 [ Jun 21 21:37:12 No 'refresh' method defined. Treating as :true. ]
6 note: configured to log to "/dev/stdout"
72024-06-21T21:37:12.027ZINFOSledAgent: Ensuring contract reaper thread file = illumos-utils/src/running_zone.rs:163
82024-06-21T21:37:12.027ZINFOSledAgent (BootstrapAgentStartup): finding links ["net0", "net1"] file = sled-agent/src/bootstrap/pre_server.rs:255
92024-06-21T21:37:12.080ZINFOSledAgent (BootstrapAgentStartup): Starting mg-ddm service addr-objs = [AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }] file = sled-agent/src/bootstrap/pre_server.rs:262
102024-06-21T21:37:12.080ZINFOSledAgent (BootstrapAgentStartup): Importing mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:42 path = /opt/oxide/mg-ddm/pkg/ddm/manifest.xml
112024-06-21T21:37:12.194ZINFOSledAgent (BootstrapAgentStartup): Setting mg-ddm interfaces file = sled-agent/src/bootstrap/maghemite.rs:50 interfaces = ("net0/ll" "net1/ll")
122024-06-21T21:37:12.203ZINFOSledAgent (BootstrapAgentStartup): Enabling mg-ddm service file = sled-agent/src/bootstrap/maghemite.rs:56
132024-06-21T21:37:12.211ZINFOSledAgent (BootstrapAgentStartup): Ensuring zfs key directory exists file = sled-agent/src/bootstrap/pre_server.rs:272 path = /var/run/oxide/
142024-06-21T21:37:12.813ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fdb0:18c0:4d0d:a02a::, len: 64 }
152024-06-21T21:37:12.955ZINFOSledAgent: Starting KeyManager file = sled-agent/src/long_running_tasks.rs:119
162024-06-21T21:37:12.955ZINFOSledAgent: Starting StorageManager file = sled-agent/src/long_running_tasks.rs:131
172024-06-21T21:37:12.955ZINFOSledAgent: Starting StorageMonitor file = sled-agent/src/long_running_tasks.rs:141
182024-06-21T21:37:12.955ZINFOSledAgent: Starting HardwareManager file = sled-agent/src/long_running_tasks.rs:162 nongimlet_observed_disks = [] sled_mode = Scrimlet { asic: SoftNpuZone }
192024-06-21T21:37:12.955ZINFOSledAgent (DumpSetup-worker): DumpSetup poll loop started. file = sled-agent/src/dump_setup.rs:506
202024-06-21T21:37:12.955ZINFOSledAgent (HardwareManager): Creating HardwareManager file = sled-hardware/src/illumos/mod.rs:625
212024-06-21T21:37:12.958ZINFOSledAgent (HardwareManager): Generated i86pc baseboard Pc { identifier: "sock", model: "i86pc" } file = sled-hardware/src/illumos/mod.rs:535
222024-06-21T21:37:12.958ZWARNSledAgent (HardwareManager): Device is not a Gimlet (i86pc), proceeding with null hardware view file = sled-hardware/src/illumos/mod.rs:674
232024-06-21T21:37:12.958ZINFOSledAgent: Starting HardwareMonitor file = sled-agent/src/long_running_tasks.rs:176
242024-06-21T21:37:12.958ZINFOSledAgent (HardwareManager): Monitoring for hardware updates file = sled-hardware/src/illumos/mod.rs:733
252024-06-21T21:37:12.958ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_0.vdev
262024-06-21T21:37:12.958ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = None
272024-06-21T21:37:12.958ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }), tx: ... } file = sled-storage/src/manager.rs:381
282024-06-21T21:37:12.958ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
292024-06-21T21:37:12.958ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } path = /scratch/m2_0.vdev
302024-06-21T21:37:12.969ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_0.vdev file = sled-hardware/src/disk.rs:304
312024-06-21T21:37:12.969ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool)
322024-06-21T21:37:13.069ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal }
332024-06-21T21:37:13.274ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
342024-06-21T21:37:13.274ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/m2_1.vdev
352024-06-21T21:37:13.274ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
362024-06-21T21:37:13.274ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
372024-06-21T21:37:13.274ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }), tx: ... } file = sled-storage/src/manager.rs:381
382024-06-21T21:37:13.274ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
392024-06-21T21:37:13.274ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } path = /scratch/m2_1.vdev
402024-06-21T21:37:13.283ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/m2_1.vdev file = sled-hardware/src/disk.rs:304
412024-06-21T21:37:13.283ZINFOSledAgent (StorageResources): Formatting zpool with generated ID file = sled-hardware/src/disk.rs:315 id = 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool)
422024-06-21T21:37:13.369ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal }
432024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_0.vdev
442024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
452024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
462024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
472024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }), tx: ... } file = sled-storage/src/manager.rs:381
482024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
492024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_1.vdev
502024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
512024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
522024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
532024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }), tx: ... } file = sled-storage/src/manager.rs:381
542024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
552024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_2.vdev
562024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }), tx: ... } file = sled-storage/src/manager.rs:381
572024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
582024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
592024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
602024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
612024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_3.vdev
622024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
632024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
642024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
652024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }), tx: ... } file = sled-storage/src/manager.rs:381
662024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
672024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_4.vdev
682024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }), tx: ... } file = sled-storage/src/manager.rs:381
692024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
702024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
712024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
722024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
732024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_5.vdev
742024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
752024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }), tx: ... } file = sled-storage/src/manager.rs:381
762024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
772024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
782024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
792024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_6.vdev
802024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }), tx: ... } file = sled-storage/src/manager.rs:381
812024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
822024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
832024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
842024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
852024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_7.vdev
862024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
872024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
882024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
892024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }), tx: ... } file = sled-storage/src/manager.rs:381
902024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
912024-06-21T21:37:13.576ZINFOSledAgent: Upserting synthetic device to Storage Manager file = sled-agent/src/long_running_tasks.rs:232 vdev = /scratch/u2_8.vdev
922024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received DetectedRawDisk { raw_disk: Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }), tx: ... } file = sled-storage/src/manager.rs:381
932024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
942024-06-21T21:37:13.576ZINFOSledAgent (StorageResources): Inserting disk file = sled-storage/src/resources.rs:494 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
952024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
962024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
972024-06-21T21:37:13.576ZINFOSledAgent: Waiting for boot disk file = sled-agent/src/long_running_tasks.rs:91
982024-06-21T21:37:13.576ZINFOSledAgent: Found boot disk DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" } file = sled-agent/src/long_running_tasks.rs:93
992024-06-21T21:37:13.576ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: Unmanaged(Synthetic(RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
1002024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
1012024-06-21T21:37:13.576ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
1022024-06-21T21:37:13.576ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1032024-06-21T21:37:13.576ZINFOSledAgent: Starting Bootstore file = sled-agent/src/long_running_tasks.rs:200
1042024-06-21T21:37:13.576ZINFOSledAgent (bootstore): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1052024-06-21T21:37:13.576ZINFOSledAgent (bootstore): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/cluster/bootstore-fsm-state.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1062024-06-21T21:37:13.576ZINFOSledAgent (bootstore): No ledger found. Loading Fsm::State as Uninitialized file = bootstore/src/schemes/v0/storage.rs:83 peer_id = pc-sock-i86pc
1072024-06-21T21:37:13.576ZINFOSledAgent (bootstore): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1082024-06-21T21:37:13.576ZINFOSledAgent (bootstore): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/cluster/bootstore-network-config.json file = common/src/ledger.rs:188 peer_id = pc-sock-i86pc
1092024-06-21T21:37:13.576ZINFOSledAgent (bootstore): No ledger found for network config file = bootstore/src/schemes/v0/storage.rs:146 peer_id = pc-sock-i86pc
1102024-06-21T21:37:13.576ZINFOSledAgent: Starting Bootstore DDMD poller file = sled-agent/src/long_running_tasks.rs:205
1112024-06-21T21:37:13.576ZINFOSledAgent: Starting ZoneBundler related tasks file = sled-agent/src/long_running_tasks.rs:220
1122024-06-21T21:37:13.587ZINFOSledAgent (ServiceManager): Creating ServiceManager file = sled-agent/src/services.rs:710
1132024-06-21T21:37:13.588ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 5194, tv_nsec: 988841726 } time_to_next_cleanup = 599.999884367s
1142024-06-21T21:37:13.588ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3547
1152024-06-21T21:37:13.588ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1162024-06-21T21:37:13.588ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/sled-agent-request.json file = common/src/ledger.rs:188
1172024-06-21T21:37:13.588ZINFOSledAgent (BootstrapAgentStartup): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/sled-agent-request.json file = common/src/ledger.rs:188
1182024-06-21T21:37:13.591ZINFOSledAgent (dropshot (BootstrapAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fdb0:18c0:4d0d:a02a::1]:80
1192024-06-21T21:37:13.591ZINFOSledAgent (SledAgentSprocketsServer): Started listening file = sled-agent/src/bootstrap/sprockets_server.rs:44 local_addr = [fdb0:18c0:4d0d:a02a::1]:12346
1202024-06-21T21:37:13.591ZINFOSledAgent (RSS): Injecting RSS configuration: RackInitializeRequest { trust_quorum_peers: None, bootstrap_discovery: OnlyOurs, ntp_servers: [ "0.pool.ntp.org", ], dns_servers: [ 1.1.1.1, 9.9.9.9, ], internal_services_ip_pool_ranges: [ V4( Ipv4Range { first: 10.151.2.164, last: 10.151.2.173, }, ), ], external_dns_ips: [ 10.151.2.164, 10.151.2.165, ], external_dns_zone_name: "oxide.test", external_certificates: "<redacted>", recovery_silo: RecoverySiloConfig { silo_name: Name( "recovery", ), user_name: UserId( "recovery", ), user_password_hash: NewPasswordHash( PasswordHashString { string: "$argon2id$v=19$m=98304,t=13,p=1$RUlWc0ZxaHo0WFdrN0N6ZQ$S8p52j85GPvMhR/ek3GL0el/oProgTwWpHJZ8lsQQoY", encoding: B64, }, ), }, rack_network_config: RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false, }, ], bgp: [], bfd: [], }, allowed_source_ips: Any, } file = sled-agent/src/rack_setup/service.rs:1011
1212024-06-21T21:37:13.591ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1222024-06-21T21:37:13.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1232024-06-21T21:37:13.591ZINFOSledAgent (RSS): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1242024-06-21T21:37:13.591ZINFOSledAgent (RSS): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/rss-plan-completed.marker file = common/src/ledger.rs:188
1252024-06-21T21:37:13.591ZINFOSledAgent (RSS): RSS configuration has not been fully applied yet file = sled-agent/src/rack_setup/service.rs:1071
1262024-06-21T21:37:13.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1272024-06-21T21:37:13.591ZINFOSledAgent (RSS): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/rss-sled-plan.json file = common/src/ledger.rs:188
1282024-06-21T21:37:13.591ZINFOSledAgent (RSS): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/rss-sled-plan.json file = common/src/ledger.rs:188
1292024-06-21T21:37:13.591ZINFOSledAgent (RSS): Creating new allocation plan file = sled-agent/src/rack_setup/service.rs:1112
1302024-06-21T21:37:13.591ZINFOSledAgent (RSS): Serializing plan file = sled-agent/src/rack_setup/plan/sled.rs:156
1312024-06-21T21:37:13.591ZINFOSledAgent (RSS): Creating plan for the sled at fdb0:18c0:4d0d:a02a::1 file = sled-agent/src/rack_setup/plan/sled.rs:129
1322024-06-21T21:37:13.591ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1332024-06-21T21:37:13.591ZINFOSledAgent (RSS): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1342024-06-21T21:37:13.592ZINFOSledAgent (RSS): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.rss-sled-plan.json.tmp file = common/src/ledger.rs:199
1352024-06-21T21:37:13.592ZINFOSledAgent (RSS): Sled plan written to storage file = sled-agent/src/rack_setup/plan/sled.rs:176
1362024-06-21T21:37:13.592ZINFOSledAgent (RSS): Writing Rack Network Configuration to bootstore file = sled-agent/src/rack_setup/service.rs:1145
1372024-06-21T21:37:13.592ZINFOSledAgent (bootstore): Attempting to update network config with generation: 1, current_generation: 0 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
1382024-06-21T21:37:13.592ZINFOSledAgent (bootstore): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1392024-06-21T21:37:13.592ZINFOSledAgent (bootstore): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
1402024-06-21T21:37:13.592ZINFOSledAgent (bootstore): Broadcasting network config with generation 1 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
1412024-06-21T21:37:13.592ZINFOSledAgent (BootstrapAgentRssHandler): Received initialization request from RSS file = sled-agent/src/bootstrap/rss_handle.rs:217 request = StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a27c09e4-c9a4-4472-b248-8c2ae9bae00f, rack_id: 884d2112-d3cd-48eb-9cc7-ecb0a29f78a9, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
1422024-06-21T21:37:13.592ZINFOSledAgent (SledAgentSprocketsServer): Accepted connection file = sled-agent/src/bootstrap/sprockets_server.rs:66 remote_addr = [fdb0:18c0:4d0d:a02a::1]:43025
1432024-06-21T21:37:13.592ZINFOSledAgent (SledAgentMain): Loading Sled Agent: StartSledAgentRequest { generation: 0, schema_version: 1, body: StartSledAgentRequestBody { id: a27c09e4-c9a4-4472-b248-8c2ae9bae00f, rack_id: 884d2112-d3cd-48eb-9cc7-ecb0a29f78a9, use_trust_quorum: false, is_lrtq_learner: false, subnet: Ipv6Subnet { net: Ipv6Net { addr: fd00:1122:3344:101::, width: 64 } } } } file = sled-agent/src/bootstrap/server.rs:360
1442024-06-21T21:37:13.592ZINFOSledAgent (SledAgentMain): KeyManager: using hardcoded secret retriever file = sled-agent/src/bootstrap/server.rs:375
1452024-06-21T21:37:13.592ZINFOSledAgent: setting up sled agent server file = sled-agent/src/server.rs:44
1462024-06-21T21:37:13.592ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
1472024-06-21T21:37:13.593ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:101::, len: 64 }
1482024-06-21T21:37:13.593ZINFOSledAgent (StorageManager): Received KeyManagerReady file = sled-storage/src/manager.rs:381
1492024-06-21T21:37:13.593ZINFOSledAgent (StorageManager): Loading storage config file = sled-storage/src/manager.rs:530
1502024-06-21T21:37:13.593ZINFOSledAgent (StorageManager): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1512024-06-21T21:37:13.593ZINFOSledAgent (StorageManager): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = load_ledger
1522024-06-21T21:37:13.593ZINFOSledAgent (StorageManager): No ledger of physical disks exists file = sled-storage/src/manager.rs:519
1532024-06-21T21:37:13.593ZINFOSledAgent (StorageManager): KeyManager ready, but no ledger detected file = sled-storage/src/manager.rs:547
1542024-06-21T21:37:13.595ZINFOSledAgent (ServiceManager): Enabling switch zone (new) file = sled-agent/src/services.rs:3779
1552024-06-21T21:37:13.605ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
1562024-06-21T21:37:13.616ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
1572024-06-21T21:37:13.625ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
1582024-06-21T21:37:13.634ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
1592024-06-21T21:37:13.642ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
1602024-06-21T21:37:13.650ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
1612024-06-21T21:37:13.659ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
1622024-06-21T21:37:13.667ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
1632024-06-21T21:37:13.676ZINFOSledAgent (StorageManager): Zpool does not exist err = ZpoolDoesNotExist file = sled-storage/src/manager.rs:603 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
1642024-06-21T21:37:13.676ZINFOSledAgent (StorageManager): No disks to be automatically managed file = sled-storage/src/manager.rs:558
1652024-06-21T21:37:13.676ZINFOSledAgent (StorageManager): automatically managing disks file = sled-storage/src/manager.rs:370
1662024-06-21T21:37:13.676ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
1672024-06-21T21:37:13.676ZINFOSledAgent (StorageManager): Successfully synchronized disks without error file = sled-storage/src/manager.rs:449 result = DisksManagementResult { status: [] }
1682024-06-21T21:37:13.676ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1692024-06-21T21:37:13.682ZINFOSledAgent: SledAgent::new(..) starting file = sled-agent/src/sled_agent.rs:354 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
1702024-06-21T21:37:13.682ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
1712024-06-21T21:37:13.682ZINFOSledAgent: Requested swap device of size 64 GiB file = sled-agent/src/sled_agent.rs:366 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
1722024-06-21T21:37:13.682ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1732024-06-21T21:37:13.687ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 50.169527ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "fcc56b15-8980-4a2f-879f-9c405d39883f", "content-length": "124", "date": "Fri, 21 Jun 2024 21:37:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "fcc56b15-8980-4a2f-879f-9c405d39883f" } file = clients/ddm-admin-client/src/lib.rs:178
1742024-06-21T21:37:13.696ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:315
1752024-06-21T21:37:13.698ZINFOSledAgent: attempting to create encrypted zvol: name="oxi_9af8bc90-90a3-45f5-b12b-12fd9904a27e/swap", size_gb=64 file = sled-agent/src/swap_device.rs:153
1762024-06-21T21:37:13.713ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_switch file = illumos-utils/src/zone.rs:354
1772024-06-21T21:37:13.722ZINFOSledAgent: successfully created encrypted zvol: name="oxi_9af8bc90-90a3-45f5-b12b-12fd9904a27e/swap", size_gb=64 file = sled-agent/src/swap_device.rs:249
1782024-06-21T21:37:13.722ZINFOSledAgent: adding swap device: swapname="/dev/zvol/dsk/oxi_9af8bc90-90a3-45f5-b12b-12fd9904a27e/swap" file = sled-agent/src/swap_device.rs:96
1792024-06-21T21:37:13.727ZINFOSledAgent: Mounting backing filesystems file = sled-agent/src/sled_agent.rs:381 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
1802024-06-21T21:37:13.727ZINFOSledAgent (BackingFs): Processing fmd file = sled-agent/src/backing_fs.rs:126
1812024-06-21T21:37:13.727ZINFOSledAgent (BackingFs): Ensuring dataset oxi_9af8bc90-90a3-45f5-b12b-12fd9904a27e/backing/fmd file = sled-agent/src/backing_fs.rs:136
1822024-06-21T21:37:13.739ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1832024-06-21T21:37:13.740ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 72.419733ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ef58e90e-1699-4e39-8ed2-16481d55b562", "content-length": "124", "date": "Fri, 21 Jun 2024 21:37:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ef58e90e-1699-4e39-8ed2-16481d55b562" } file = clients/ddm-admin-client/src/lib.rs:178
1842024-06-21T21:37:13.798ZINFOSledAgent (BackingFs): Stopping service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:170
1852024-06-21T21:37:13.813ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1862024-06-21T21:37:13.813ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 158.994459ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a71e2d0b-97e4-4a6f-9fee-8c4ac5ba407d", "content-length": "124", "date": "Fri, 21 Jun 2024 21:37:12 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a71e2d0b-97e4-4a6f-9fee-8c4ac5ba407d" } file = clients/ddm-admin-client/src/lib.rs:178
1872024-06-21T21:37:13.973ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1882024-06-21T21:37:13.974ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 185.989247ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "46c44140-8a9a-41e4-b3c0-2c4914d77b4c", "content-length": "124", "date": "Fri, 21 Jun 2024 21:37:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "46c44140-8a9a-41e4-b3c0-2c4914d77b4c" } file = clients/ddm-admin-client/src/lib.rs:178
1892024-06-21T21:37:14.017ZINFOSledAgent (BackingFs): Mounting oxi_9af8bc90-90a3-45f5-b12b-12fd9904a27e/backing/fmd on /var/fm/fmd file = sled-agent/src/backing_fs.rs:178
1902024-06-21T21:37:14.030ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/rsrc file = sled-agent/src/backing_fs.rs:186
1912024-06-21T21:37:14.031ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/ckpt file = sled-agent/src/backing_fs.rs:186
1922024-06-21T21:37:14.031ZINFOSledAgent (BackingFs): Ensuring directory /var/fm/fmd/xprt file = sled-agent/src/backing_fs.rs:186
1932024-06-21T21:37:14.031ZINFOSledAgent (BackingFs): Starting service svc:/system/fmd:default file = sled-agent/src/backing_fs.rs:192
1942024-06-21T21:37:14.161ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1952024-06-21T21:37:14.161ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 302.25226ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "38768002-2e98-4f44-99b0-9243fd62e02a", "content-length": "124", "date": "Fri, 21 Jun 2024 21:37:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "38768002-2e98-4f44-99b0-9243fd62e02a" } file = clients/ddm-admin-client/src/lib.rs:178
1962024-06-21T21:37:14.465ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
1972024-06-21T21:37:14.465ZINFOSledAgent (BootstrapAgentStartup): Failed enable ddm stats (will retry after 329.669884ms DdmAdminClient = [::1]:8000 err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4bdd07c6-2984-4105-9e90-e5c30511e0fa", "content-length": "124", "date": "Fri, 21 Jun 2024 21:37:13 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4bdd07c6-2984-4105-9e90-e5c30511e0fa" } file = clients/ddm-admin-client/src/lib.rs:178
1982024-06-21T21:37:14.725ZINFOSledAgent: using '[AddrObject { interface: "net0", name: "ll" }, AddrObject { interface: "net1", name: "ll" }]' as data links for xde driver file = illumos-utils/src/opte/illumos.rs:90 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
1992024-06-21T21:37:14.726ZINFOSledAgent (dropshot): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:44452 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2002024-06-21T21:37:14.726ZINFOSledAgent (internal-dns-resolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2012024-06-21T21:37:14.726ZINFOSledAgent (producer-server): starting oximeter metric producer server address = [fd00:1122:3344:101::1]:44452 file = oximeter/producer/src/lib.rs:276 interval = 30s producer_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2022024-06-21T21:37:14.727ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 141.191171ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2032024-06-21T21:37:14.772ZINFOSledAgent (VmmReservoirManager): 50% of 102952026112 physical ram = 51476013056 bytes) file = sled-agent/src/vmm_reservoir.rs:267 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2042024-06-21T21:37:14.772ZINFOSledAgent (VmmReservoirManager): Setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:275 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2052024-06-21T21:37:14.772ZINFOSledAgent (ServiceManager): sled agent started file = sled-agent/src/services.rs:893 underlay_address = fd00:1122:3344:101::1
2062024-06-21T21:37:14.772ZINFOSledAgent (DnsResolver): new DNS resolver addresses = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = internal-dns/src/resolver.rs:60
2072024-06-21T21:37:14.773ZINFOSledAgent (dropshot (SledAgent)): listening file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:196 local_addr = [fd00:1122:3344:101::1]:12345
2082024-06-21T21:37:14.773ZINFOSledAgent (SledAgentMain): Sled Agent loaded; recording configuration file = sled-agent/src/bootstrap/server.rs:430
2092024-06-21T21:37:14.773ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2102024-06-21T21:37:14.773ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2112024-06-21T21:37:14.773ZINFOSledAgent (SledAgentMain): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.sled-agent-request.json.tmp file = common/src/ledger.rs:199
2122024-06-21T21:37:14.773ZINFOSledAgent (HardwareMonitor): Sled Agent Started file = sled-agent/src/hardware_monitor.rs:138
2132024-06-21T21:37:14.773ZINFOSledAgent (HardwareMonitor): Checking current full hardware snapshot disks = {} file = sled-agent/src/hardware_monitor.rs:262 underlay_network_info = Some((fd00:1122:3344:101::2, Some(RackNetworkConfigV2 { rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56 }, infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [PortConfigV2 { routes: [RouteConfig { destination: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), nexthop: 10.151.2.100, vlan_id: None }], addresses: [UplinkAddressConfig { address: V4(Ipv4Net { addr: 10.151.2.174, width: 24 }), vlan_id: None }], switch: Switch0, port: "qsfp0", uplink_port_speed: Speed40G, uplink_port_fec: None, bgp_peers: [], autoneg: false }], bgp: [], bfd: [] })))
2142024-06-21T21:37:14.773ZINFOSledAgent (ServiceManager): Ensuring scrimlet services (enabling services) file = sled-agent/src/services.rs:3547
2152024-06-21T21:37:14.773ZINFOSledAgent (SledAgentSprocketsServer): Connection closed file = sled-agent/src/bootstrap/sprockets_server.rs:73 remote_addr = [fdb0:18c0:4d0d:a02a::1]:43025
2162024-06-21T21:37:14.949ZINFOSledAgent (BootstrapAgentRssHandler): Peer agent initialized file = sled-agent/src/bootstrap/rss_handle.rs:108 peer = [fdb0:18c0:4d0d:a02a::1]:12346
2172024-06-21T21:37:14.949ZINFOSledAgent (BootstrapAgentRssHandler): Initialized sled agent file = sled-agent/src/bootstrap/rss_handle.rs:232 target_sled = [fdb0:18c0:4d0d:a02a::1]:12346
2182024-06-21T21:37:14.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2192024-06-21T21:37:14.949ZINFOSledAgent (RSS): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2202024-06-21T21:37:14.949ZINFOSledAgent (RSS): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/rss-service-plan-v3.json file = common/src/ledger.rs:188
2212024-06-21T21:37:14.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2222024-06-21T21:37:14.949ZINFOSledAgent (BootstrapAgentStartup): Enabling ddm stats DdmAdminClient = [::1]:8000 addr = fd00:1122:3344:101::1 dns_servers = [[fd00:1122:3344:1::1]:53, [fd00:1122:3344:2::1]:53, [fd00:1122:3344:3::1]:53, [fd00:1122:3344:4::1]:53, [fd00:1122:3344:5::1]:53] file = clients/ddm-admin-client/src/lib.rs:164
2232024-06-21T21:37:14.949ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2242024-06-21T21:37:14.950ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 671.029377ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2252024-06-21T21:37:14.955ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2262024-06-21T21:37:14.987ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40461
2272024-06-21T21:37:14.987ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2282024-06-21T21:37:14.987ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 262 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40461 req_id = beb9e2fc-4196-40cc-b2a0-5348475fa272 response_code = 200 uri = /inventory
2292024-06-21T21:37:15.024ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:41363
2302024-06-21T21:37:15.025ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 37 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:41363 req_id = 6041313a-94c3-4a51-ba58-9d0f294dce29 response_code = 200 uri = /sled-role
2312024-06-21T21:37:15.026ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
2322024-06-21T21:37:15.026ZINFOSledAgent (RSS): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2332024-06-21T21:37:15.027ZINFOSledAgent (RSS): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.rss-service-plan-v3.json.tmp file = common/src/ledger.rs:199
2342024-06-21T21:37:15.027ZINFOSledAgent (RSS): Service plan written to storage file = sled-agent/src/rack_setup/plan/service.rs:812
2352024-06-21T21:37:15.058ZINFOSledAgent (RSS): attempting to set up sled's storage: OmicronPhysicalDisksConfig { disks: [OmicronPhysicalDiskConfig { id: 6cd081a2-283c-43a2-a300-b0899484b0ba, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, pool_id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool) }, OmicronPhysicalDiskConfig { id: 91eb1bb1-3bbf-4c49-884a-85d7a34c2137, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, pool_id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool) }, OmicronPhysicalDiskConfig { id: 83b1edcd-e5df-45c2-b7a3-53c029865fba, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, pool_id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool) }, OmicronPhysicalDiskConfig { id: 6799c5cc-a2dd-4cd8-8455-656a136c2ca0, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, pool_id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool) }, OmicronPhysicalDiskConfig { id: e7e5d64e-5ccb-4079-a9eb-dd436ad5ea7e, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, pool_id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool) }, OmicronPhysicalDiskConfig { id: 3fe012df-50e6-4b55-b0c8-2345a041579e, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, pool_id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool) }, OmicronPhysicalDiskConfig { id: ef750f3b-5c0e-43a6-985a-c9de50bba937, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, pool_id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool) }, OmicronPhysicalDiskConfig { id: 755c2842-c938-497b-b846-6141fa1bfdc4, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, pool_id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool) }, OmicronPhysicalDiskConfig { id: 7b8511c8-ea59-4a7f-bfe7-3d59a7200e97, identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, pool_id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool) }], generation: Generation(1) } file = sled-agent/src/rack_setup/service.rs:348 sled_address = [fd00:1122:3344:101::1]:12345
2362024-06-21T21:37:15.058ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57235
2372024-06-21T21:37:15.059ZINFOSledAgent (StorageManager): Received OmicronPhysicalDisksEnsure { config: OmicronPhysicalDisksConfig { generation: Generation(1), disks: [OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, id: 6cd081a2-283c-43a2-a300-b0899484b0ba, pool_id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, id: 91eb1bb1-3bbf-4c49-884a-85d7a34c2137, pool_id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, id: 83b1edcd-e5df-45c2-b7a3-53c029865fba, pool_id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, id: 6799c5cc-a2dd-4cd8-8455-656a136c2ca0, pool_id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, id: e7e5d64e-5ccb-4079-a9eb-dd436ad5ea7e, pool_id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, id: 3fe012df-50e6-4b55-b0c8-2345a041579e, pool_id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, id: ef750f3b-5c0e-43a6-985a-c9de50bba937, pool_id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, id: 755c2842-c938-497b-b846-6141fa1bfdc4, pool_id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool) }, OmicronPhysicalDiskConfig { identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, id: 7b8511c8-ea59-4a7f-bfe7-3d59a7200e97, pool_id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool) }] }, tx: ... } file = sled-storage/src/manager.rs:381
2382024-06-21T21:37:15.059ZINFOSledAgent (StorageManager): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2392024-06-21T21:37:15.059ZINFOSledAgent (StorageManager): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-physical-disks.json file = common/src/ledger.rs:188 request = omicron_physical_disks_ensure
2402024-06-21T21:37:15.059ZINFOSledAgent (StorageManager): No previously-stored 'requested disks', creating new ledger file = sled-storage/src/manager.rs:681 request = omicron_physical_disks_ensure
2412024-06-21T21:37:15.059ZINFOSledAgent (StorageResources): Synchronizing disk managment file = sled-storage/src/resources.rs:339
2422024-06-21T21:37:15.059ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:379
2432024-06-21T21:37:15.059ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2442024-06-21T21:37:15.059ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } path = /scratch/u2_0.vdev
2452024-06-21T21:37:15.169ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_0.vdev file = sled-hardware/src/disk.rs:304
2462024-06-21T21:37:15.169ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 5b452253-5f6d-4617-91b5-3f263edef77d (zpool)
2472024-06-21T21:37:15.699ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 918.594688ms error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2482024-06-21T21:37:16.113ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2492024-06-21T21:37:16.122ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }
2502024-06-21T21:37:16.289ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/dataset.rs:332
2512024-06-21T21:37:16.290ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:334
2522024-06-21T21:37:16.290ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:338
2532024-06-21T21:37:16.290ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } epoch = 0 file = sled-storage/src/dataset.rs:340
2542024-06-21T21:37:16.290ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2552024-06-21T21:37:16.290ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key")
2562024-06-21T21:37:16.290ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2572024-06-21T21:37:16.647ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.14468709s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2582024-06-21T21:37:16.822ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-0-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2592024-06-21T21:37:17.011ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_switch
2602024-06-21T21:37:17.170ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_5b452253-5f6d-4617-91b5-3f263edef77d/crypt/zone file = sled-storage/src/dataset.rs:395
2612024-06-21T21:37:17.370ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2622024-06-21T21:37:17.370ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2632024-06-21T21:37:17.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 73.468193ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2642024-06-21T21:37:18.391ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2652024-06-21T21:37:18.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 77.02865ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2662024-06-21T21:37:19.002ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 261.707535ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2672024-06-21T21:37:19.003ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 2.147715751s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2682024-06-21T21:37:19.180ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2692024-06-21T21:37:19.825ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2702024-06-21T21:37:19.825ZWARNSledAgent (ServiceManager): wait for service Some("oxz_switch") failed: Property not found. retry in 467.418799ms file = illumos-utils/src/svc.rs:36 zone = oxz_switch
2712024-06-21T21:37:20.067ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_5b452253-5f6d-4617-91b5-3f263edef77d file = sled-storage/src/dataset.rs:480
2722024-06-21T21:37:20.293ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:477
2732024-06-21T21:37:20.293ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" } file = sled-storage/src/resources.rs:394
2742024-06-21T21:37:20.293ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:379
2752024-06-21T21:37:20.293ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
2762024-06-21T21:37:20.293ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } path = /scratch/u2_1.vdev
2772024-06-21T21:37:20.511ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_1.vdev file = sled-hardware/src/disk.rs:304
2782024-06-21T21:37:20.511ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool)
2792024-06-21T21:37:21.662ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }
2802024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/dataset.rs:332
2812024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:334
2822024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:338
2832024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } epoch = 0 file = sled-storage/src/dataset.rs:340
2842024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2852024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key")
2862024-06-21T21:37:21.889ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
2872024-06-21T21:37:22.640ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2882024-06-21T21:37:22.640ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2892024-06-21T21:37:22.640ZWARNSledAgent (producer-registration-task): failed to lookup Nexus IP, will retry delay = 10.549692053s error = "proto error: io error: No route to host (os error 148)" file = oximeter/producer/src/lib.rs:384 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2902024-06-21T21:37:23.362ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-1-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
2912024-06-21T21:37:23.830ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_41b82b1f-5def-467c-bc86-5c87195bd788/crypt/zone file = sled-storage/src/dataset.rs:395
2922024-06-21T21:37:24.309ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2932024-06-21T21:37:24.309ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2942024-06-21T21:37:24.539ZINFOSledAgent (ServiceManager): Ensuring tfportrear0_0/ll exists in zone file = sled-agent/src/services.rs:2233
2952024-06-21T21:37:25.025ZINFOSledAgent (VmmReservoirManager): Finished setting reservoir size to 49090 MiB bytes file = sled-agent/src/vmm_reservoir.rs:279 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
2962024-06-21T21:37:25.137ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_41b82b1f-5def-467c-bc86-5c87195bd788 file = sled-storage/src/dataset.rs:480
2972024-06-21T21:37:25.145ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:477
2982024-06-21T21:37:25.145ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" } file = sled-storage/src/resources.rs:394
2992024-06-21T21:37:25.145ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:379
3002024-06-21T21:37:25.145ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3012024-06-21T21:37:25.145ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } path = /scratch/u2_2.vdev
3022024-06-21T21:37:25.155ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_2.vdev file = sled-hardware/src/disk.rs:304
3032024-06-21T21:37:25.155ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool)
3042024-06-21T21:37:25.250ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }
3052024-06-21T21:37:25.258ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/dataset.rs:332
3062024-06-21T21:37:25.258ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:334
3072024-06-21T21:37:25.258ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:338
3082024-06-21T21:37:25.258ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } epoch = 0 file = sled-storage/src/dataset.rs:340
3092024-06-21T21:37:25.258ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3102024-06-21T21:37:25.258ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key")
3112024-06-21T21:37:25.259ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3122024-06-21T21:37:25.294ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-2-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3132024-06-21T21:37:25.302ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_44b9c17b-6e37-45fe-b9da-6182ccbe59a4/crypt/zone file = sled-storage/src/dataset.rs:395
3142024-06-21T21:37:25.313ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
3152024-06-21T21:37:25.426ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_44b9c17b-6e37-45fe-b9da-6182ccbe59a4 file = sled-storage/src/dataset.rs:480
3162024-06-21T21:37:25.434ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:477
3172024-06-21T21:37:25.434ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" } file = sled-storage/src/resources.rs:394
3182024-06-21T21:37:25.434ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:379
3192024-06-21T21:37:25.434ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3202024-06-21T21:37:25.434ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } path = /scratch/u2_3.vdev
3212024-06-21T21:37:25.444ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_3.vdev file = sled-hardware/src/disk.rs:304
3222024-06-21T21:37:25.444ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool)
3232024-06-21T21:37:25.534ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }
3242024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/dataset.rs:332
3252024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:334
3262024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:338
3272024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } epoch = 0 file = sled-storage/src/dataset.rs:340
3282024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3292024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key")
3302024-06-21T21:37:25.543ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3312024-06-21T21:37:25.581ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-3-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3322024-06-21T21:37:25.590ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_e6f1c1a1-ab63-45ad-a43f-153910e0494a/crypt/zone file = sled-storage/src/dataset.rs:395
3332024-06-21T21:37:25.712ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_e6f1c1a1-ab63-45ad-a43f-153910e0494a file = sled-storage/src/dataset.rs:480
3342024-06-21T21:37:25.720ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:477
3352024-06-21T21:37:25.720ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" } file = sled-storage/src/resources.rs:394
3362024-06-21T21:37:25.720ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:379
3372024-06-21T21:37:25.720ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3382024-06-21T21:37:25.720ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } path = /scratch/u2_4.vdev
3392024-06-21T21:37:25.730ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_4.vdev file = sled-hardware/src/disk.rs:304
3402024-06-21T21:37:25.730ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool)
3412024-06-21T21:37:25.824ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }
3422024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/dataset.rs:332
3432024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:334
3442024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:338
3452024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } epoch = 0 file = sled-storage/src/dataset.rs:340
3462024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3472024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key")
3482024-06-21T21:37:25.832ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3492024-06-21T21:37:25.867ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-4-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3502024-06-21T21:37:25.875ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_8ce4d8fb-4017-46e3-ac55-3c40125a531b/crypt/zone file = sled-storage/src/dataset.rs:395
3512024-06-21T21:37:25.993ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_8ce4d8fb-4017-46e3-ac55-3c40125a531b file = sled-storage/src/dataset.rs:480
3522024-06-21T21:37:26.000ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:477
3532024-06-21T21:37:26.000ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" } file = sled-storage/src/resources.rs:394
3542024-06-21T21:37:26.000ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:379
3552024-06-21T21:37:26.000ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3562024-06-21T21:37:26.000ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } path = /scratch/u2_5.vdev
3572024-06-21T21:37:26.010ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_5.vdev file = sled-hardware/src/disk.rs:304
3582024-06-21T21:37:26.010ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool)
3592024-06-21T21:37:26.101ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }
3602024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/dataset.rs:332
3612024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:334
3622024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:338
3632024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } epoch = 0 file = sled-storage/src/dataset.rs:340
3642024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3652024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key")
3662024-06-21T21:37:26.109ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3672024-06-21T21:37:26.113ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
3682024-06-21T21:37:26.144ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-5-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3692024-06-21T21:37:26.152ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_a0c5f5f0-1cc5-45cc-9434-17b4abf3b972/crypt/zone file = sled-storage/src/dataset.rs:395
3702024-06-21T21:37:26.278ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 file = sled-storage/src/dataset.rs:480
3712024-06-21T21:37:26.285ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:477
3722024-06-21T21:37:26.285ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" } file = sled-storage/src/resources.rs:394
3732024-06-21T21:37:26.286ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:379
3742024-06-21T21:37:26.286ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3752024-06-21T21:37:26.286ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } path = /scratch/u2_6.vdev
3762024-06-21T21:37:26.295ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_6.vdev file = sled-hardware/src/disk.rs:304
3772024-06-21T21:37:26.295ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool)
3782024-06-21T21:37:26.392ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }
3792024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/dataset.rs:332
3802024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:334
3812024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:338
3822024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } epoch = 0 file = sled-storage/src/dataset.rs:340
3832024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3842024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key")
3852024-06-21T21:37:26.400ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
3862024-06-21T21:37:26.405ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
3872024-06-21T21:37:26.437ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-6-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
3882024-06-21T21:37:26.446ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_85a7fd4f-88d4-408c-a832-0d457152c72f/crypt/zone file = sled-storage/src/dataset.rs:395
3892024-06-21T21:37:26.571ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_85a7fd4f-88d4-408c-a832-0d457152c72f file = sled-storage/src/dataset.rs:480
3902024-06-21T21:37:26.579ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:477
3912024-06-21T21:37:26.579ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" } file = sled-storage/src/resources.rs:394
3922024-06-21T21:37:26.579ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:379
3932024-06-21T21:37:26.579ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
3942024-06-21T21:37:26.579ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } path = /scratch/u2_7.vdev
3952024-06-21T21:37:26.590ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_7.vdev file = sled-hardware/src/disk.rs:304
3962024-06-21T21:37:26.590ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool)
3972024-06-21T21:37:26.691ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }
3982024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/dataset.rs:332
3992024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:334
4002024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:338
4012024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } epoch = 0 file = sled-storage/src/dataset.rs:340
4022024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4032024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key")
4042024-06-21T21:37:26.699ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4052024-06-21T21:37:26.737ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-7-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4062024-06-21T21:37:26.745ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_6b2e5449-891f-4dfd-8e2d-78189eabe443/crypt/zone file = sled-storage/src/dataset.rs:395
4072024-06-21T21:37:26.868ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_6b2e5449-891f-4dfd-8e2d-78189eabe443 file = sled-storage/src/dataset.rs:480
4082024-06-21T21:37:26.876ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:477
4092024-06-21T21:37:26.876ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" } file = sled-storage/src/resources.rs:394
4102024-06-21T21:37:26.876ZINFOSledAgent (StorageResources): Managing disk disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:379
4112024-06-21T21:37:26.876ZINFOSledAgent (StorageResources): Invoking Disk::new on an unmanaged disk file = sled-storage/src/resources.rs:456
4122024-06-21T21:37:26.876ZINFOSledAgent (StorageResources): Invoking SyntheticDisk::new file = sled-storage/src/disk.rs:121 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } path = /scratch/u2_8.vdev
4132024-06-21T21:37:26.886ZINFOSledAgent (StorageResources): GPT exists without Zpool: formatting zpool at /scratch/u2_8.vdev file = sled-hardware/src/disk.rs:304
4142024-06-21T21:37:26.886ZINFOSledAgent (StorageResources): Formatting zpool with requested ID file = sled-hardware/src/disk.rs:310 id = 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool)
4152024-06-21T21:37:26.974ZINFOSledAgent (StorageResources): Ensuring zpool has datasets disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:288 zpool = ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }
4162024-06-21T21:37:26.983ZINFOSledAgent (StorageResources): Loading latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/dataset.rs:332
4172024-06-21T21:37:26.983ZINFOSledAgent (StorageResources): Loaded latest secret disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:334
4182024-06-21T21:37:26.983ZINFOSledAgent (StorageResources): Retrieving key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:338
4192024-06-21T21:37:26.983ZINFOSledAgent (StorageResources): Got key disk_id = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } epoch = 0 file = sled-storage/src/dataset.rs:340
4202024-06-21T21:37:26.983ZINFOSledAgent (StorageResources): About to create keyfile file = sled-storage/src/keyfile.rs:30 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4212024-06-21T21:37:26.984ZINFOSledAgent (StorageResources): Created keyfile file = sled-storage/src/keyfile.rs:34 path = Keypath("/var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key")
4222024-06-21T21:37:26.984ZINFOSledAgent (StorageResources): Ensuring encrypted filesystem: crypt for epoch 0 file = sled-storage/src/dataset.rs:352
4232024-06-21T21:37:27.019ZINFOSledAgent (StorageResources): Zeroed and unlinked keyfile /var/run/oxide/synthetic-vendor-synthetic-serial-8-synthetic-model-U2-zfs-aes-256-gcm.key file = sled-storage/src/keyfile.rs:53
4242024-06-21T21:37:27.027ZINFOSledAgent (StorageResources): Automatically destroying dataset: oxp_3ae04f9e-dd4d-4980-8e86-31c81824c907/crypt/zone file = sled-storage/src/dataset.rs:395
4252024-06-21T21:37:27.150ZINFOSledAgent (StorageResources): Looking for unencrypted datasets in oxp_3ae04f9e-dd4d-4980-8e86-31c81824c907 file = sled-storage/src/dataset.rs:480
4262024-06-21T21:37:27.158ZINFOSledAgent (StorageResources): Disk::new completed successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:477
4272024-06-21T21:37:27.158ZINFOSledAgent (StorageResources): Disk management started successfully disk_identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" } file = sled-storage/src/resources.rs:394
4282024-06-21T21:37:27.158ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4292024-06-21T21:37:27.158ZINFOSledAgent (StorageMonitor): Received storage manager update disks = AllDisks { values: {DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }, variant: M2, slot: 1024 }, zpool_name: ZpoolName { id: 9af8bc90-90a3-45f5-b12b-12fd9904a27e (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }: ImplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/m2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }, variant: M2, slot: 1025 }, zpool_name: ZpoolName { id: 89c103d3-b501-4e1d-8d78-ad9e833f154c (zpool), kind: Internal } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_0.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }, variant: U2, slot: 1026 }, zpool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_1.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }, variant: U2, slot: 1027 }, zpool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_2.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }, variant: U2, slot: 1028 }, zpool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_3.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }, variant: U2, slot: 1029 }, zpool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_4.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }, variant: U2, slot: 1030 }, zpool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_5.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }, variant: U2, slot: 1031 }, zpool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_6.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }, variant: U2, slot: 1032 }, zpool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_7.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }, variant: U2, slot: 1033 }, zpool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } })), DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }: ExplicitlyManaged(Synthetic(SyntheticDisk { raw: RawSyntheticDisk { path: "/scratch/u2_8.vdev", identity: DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }, variant: U2, slot: 1034 }, zpool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } }))}, mount_config: MountConfig { root: "/", synthetic_disk_root: "/var/tmp" } } file = sled-agent/src/storage_monitor.rs:41
4302024-06-21T21:37:27.158ZINFOSledAgent (DumpSetup-worker): Updated view of disks core_datasets = 0 debug_datasets = 0 dump_slices = 0 file = sled-agent/src/dump_setup.rs:549
4312024-06-21T21:37:27.158ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
4322024-06-21T21:37:27.158ZINFOSledAgent (StorageManager): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.omicron-physical-disks.json.tmp file = common/src/ledger.rs:199 request = omicron_physical_disks_ensure
4332024-06-21T21:37:27.159ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 12100205 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:57235 req_id = 570044f0-3af9-488c-a59d-118808de7c51 response_code = 200 uri = /omicron-physical-disks
4342024-06-21T21:37:27.188ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(2), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
4352024-06-21T21:37:27.188ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49431
4362024-06-21T21:37:27.189ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4372024-06-21T21:37:27.189ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
4382024-06-21T21:37:27.235ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4392024-06-21T21:37:27.235ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4402024-06-21T21:37:27.284ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
4412024-06-21T21:37:27.284ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
4422024-06-21T21:37:27.334ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4432024-06-21T21:37:27.334ZINFOSledAgent (ServiceManager): No ledger in /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:188
4442024-06-21T21:37:27.334ZINFOSledAgent (ServiceManager): No ledger in /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:188
4452024-06-21T21:37:27.334ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4462024-06-21T21:37:27.379ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4472024-06-21T21:37:27.379ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4482024-06-21T21:37:27.379ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
4492024-06-21T21:37:27.408ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
4502024-06-21T21:37:27.432ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb file = illumos-utils/src/zone.rs:315
4512024-06-21T21:37:27.436ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de file = illumos-utils/src/zone.rs:315
4522024-06-21T21:37:27.440ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82 file = illumos-utils/src/zone.rs:315
4532024-06-21T21:37:27.453ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb file = illumos-utils/src/zone.rs:354
4542024-06-21T21:37:27.458ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de file = illumos-utils/src/zone.rs:354
4552024-06-21T21:37:27.461ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82 file = illumos-utils/src/zone.rs:354
4562024-06-21T21:37:28.111ZWARNSledAgent (NexusNotifierTask): Received Error from Nexus for Get request: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/sled-agents/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/nexus.rs:619 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
4572024-06-21T21:37:28.411ZERROSledAgent (ProbeManager): get target probe state: Communication Error: error sending request for url (http://_nexus._tcp.control-plane.oxide.internal:12221/probes/a27c09e4-c9a4-4472-b248-8c2ae9bae00f): error trying to connect: dns error: proto error: io error: No route to host (os error 148) file = sled-agent/src/probe_manager.rs:169 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
4582024-06-21T21:37:28.980ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService1"/> <propval type="astring" name="gateway" value="fd00:1122:3344:1::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:1::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:1::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:1::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4592024-06-21T21:37:28.980ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:1::, len: 64 }
4602024-06-21T21:37:28.980ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4612024-06-21T21:37:29.469ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService2"/> <propval type="astring" name="gateway" value="fd00:1122:3344:3::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:3::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:3::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:3::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4622024-06-21T21:37:29.947ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:3::, len: 64 }
4632024-06-21T21:37:29.947ZINFOSledAgent (ServiceManager): Profile for oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService3"/> <propval type="astring" name="gateway" value="fd00:1122:3344:2::2"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:2::1"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/internal_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:2::1]:5353"/> <propval type="astring" name="dns_address" value="[fd00:1122:3344:2::1]:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
4642024-06-21T21:37:29.947ZINFOSledAgent (BootstrapAgentStartup): Sending prefix to ddmd for advertisement DdmAdminClient = [::1]:8000 file = clients/ddm-admin-client/src/lib.rs:87 prefix = Ipv6Prefix { addr: fd00:1122:3344:2::, len: 64 }
4652024-06-21T21:37:29.955ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 50.685439ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4662024-06-21T21:37:29.955ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4672024-06-21T21:37:29.959ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4682024-06-21T21:37:30.016ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 119.184165ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4692024-06-21T21:37:30.083ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 71.312941ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4702024-06-21T21:37:30.090ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 56.596795ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4712024-06-21T21:37:30.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 287.174673ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4722024-06-21T21:37:30.216ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 75.885434ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4732024-06-21T21:37:30.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 101.662273ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4742024-06-21T21:37:30.394ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 189.317966ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4752024-06-21T21:37:30.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 238.702361ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4762024-06-21T21:37:30.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 351.500805ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4772024-06-21T21:37:30.599ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 403.17341ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4782024-06-21T21:37:30.661ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 281.386479ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4792024-06-21T21:37:30.823ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 471.529661ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4802024-06-21T21:37:30.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 763.456403ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4812024-06-21T21:37:31.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 1.182839192s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4822024-06-21T21:37:31.303ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 874.892563ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4832024-06-21T21:37:31.725ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 502.779466ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4842024-06-21T21:37:32.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 939.261504ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4852024-06-21T21:37:32.204ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 1.402023492s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4862024-06-21T21:37:32.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 745.206801ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4872024-06-21T21:37:32.618ZINFOSledAgent (ServiceManager): Ensuring bootstrap address fdb0:18c0:4d0d:a02a::2 exists in switch zone file = sled-agent/src/services.rs:2250
4882024-06-21T21:37:32.619ZINFOSledAgent (ServiceManager): Adding bootstrap address file = illumos-utils/src/running_zone.rs:626 zone = oxz_switch
4892024-06-21T21:37:32.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 566.231968ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4902024-06-21T21:37:33.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 739.967891ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4912024-06-21T21:37:33.173ZINFOSledAgent (ServiceManager): Forwarding bootstrap traffic via oxBootstrap0 to fe80::8:20ff:fee5:dd9c file = sled-agent/src/services.rs:2257
4922024-06-21T21:37:33.179ZINFOSledAgent (ServiceManager): Zone not using gateway (even though underlay is up) file = sled-agent/src/services.rs:2318
4932024-06-21T21:37:33.300ZINFOSledAgent (ServiceManager): Setting up dendrite service file = sled-agent/src/services.rs:2503
4942024-06-21T21:37:33.532ZINFOSledAgent (ServiceManager): Setting up lldpd service file = sled-agent/src/services.rs:2667
4952024-06-21T21:37:33.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 1.495729466s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
4962024-06-21T21:37:33.615ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 736.212747ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
4972024-06-21T21:37:33.715ZINFOSledAgent (ServiceManager): Setting up MGS service file = sled-agent/src/services.rs:2382
4982024-06-21T21:37:33.884ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 1.094148046s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
4992024-06-21T21:37:34.006ZINFOSledAgent (ServiceManager): Setting up wicketd service file = sled-agent/src/services.rs:2417
5002024-06-21T21:37:34.203ZINFOSledAgent (ServiceManager): Setting up mgd service file = sled-agent/src/services.rs:2717
5012024-06-21T21:37:34.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 1.446389613s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
5022024-06-21T21:37:34.373ZINFOSledAgent (ServiceManager): Setting up mg-ddm service file = sled-agent/src/services.rs:2744
5032024-06-21T21:37:34.579ZINFOSledAgent (ServiceManager): Setting up tfport service file = sled-agent/src/services.rs:2611
5042024-06-21T21:37:34.753ZINFOSledAgent (ServiceManager): Setting up Simulated SP service file = sled-agent/src/services.rs:2411
5052024-06-21T21:37:34.763ZINFOSledAgent (ServiceManager): Re-enabling running switch zone (new address) file = sled-agent/src/services.rs:3803 new = [fd00:1122:3344:101::2, ::1] old = [::1]
5062024-06-21T21:37:34.763ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists file = sled-agent/src/services.rs:3819
5072024-06-21T21:37:34.763ZINFOSledAgent (ServiceManager): Adding address: Static(V6(Ipv6Network { addr: fd00:1122:3344:101::2, prefix: 64 })) file = illumos-utils/src/running_zone.rs:609 zone = oxz_switch
5082024-06-21T21:37:34.986ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb") failed: Property not found. retry in 586.307275ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
5092024-06-21T21:37:35.071ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 582.518643ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
5102024-06-21T21:37:35.311ZINFOSledAgent (ServiceManager): Ensuring address fd00:1122:3344:101::2 exists - OK file = sled-agent/src/services.rs:3827
5112024-06-21T21:37:35.317ZINFOSledAgent (ServiceManager): configuring dendrite service file = sled-agent/src/services.rs:3883
5122024-06-21T21:37:35.449ZINFOSledAgent (ServiceManager): configuring lldp service file = sled-agent/src/services.rs:3945
5132024-06-21T21:37:35.543ZINFOSledAgent (ServiceManager): configuring wicketd file = sled-agent/src/services.rs:3926 rack_subnet = fd00:1122:3344::
5142024-06-21T21:37:35.566ZINFOSledAgent (ServiceManager): configuring mgd service file = sled-agent/src/services.rs:3973
5152024-06-21T21:37:35.662ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82") failed: Property not found. retry in 903.036866ms file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
5162024-06-21T21:37:35.669ZINFOSledAgent (ServiceManager): configuring mg-ddm service file = sled-agent/src/services.rs:4000
5172024-06-21T21:37:35.792ZINFOSledAgent (ServiceManager): Determining physical location of our switch zone at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:369
5182024-06-21T21:37:35.817ZWARNSledAgent (ServiceManager): wait for service Some("oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de") failed: Property not found. retry in 1.344485721s file = illumos-utils/src/svc.rs:36 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
5192024-06-21T21:37:35.828ZINFOSledAgent (ServiceManager): Initializing 1 Uplinks on Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:417
5202024-06-21T21:37:35.860ZINFOSledAgent (ServiceManager): Building Port Configuration file = sled-agent/src/bootstrap/early_networking.rs:656
5212024-06-21T21:37:35.860ZINFOSledAgent (ServiceManager): Checking dendrite uptime file = sled-agent/src/bootstrap/early_networking.rs:694
5222024-06-21T21:37:35.860ZINFOSledAgent (ServiceManager): Dendrite online file = sled-agent/src/bootstrap/early_networking.rs:697 uptime = 2
5232024-06-21T21:37:35.860ZINFOSledAgent (ServiceManager): Configuring default uplink on switch config = PortSettings {\n links: {\n "0": LinkSettings {\n addrs: [\n 10.151.2.174,\n ],\n params: LinkCreate {\n autoneg: false,\n fec: None,\n kr: false,\n lane: Some(\n LinkId(\n 0,\n ),\n ),\n speed: Speed40G,\n },\n },\n },\n} file = sled-agent/src/bootstrap/early_networking.rs:439
5242024-06-21T21:37:35.940ZINFOSledAgent (StorageManager): Received DetectedRawDisksChanged { raw_disks: {}, tx: ... } file = sled-storage/src/manager.rs:381
5252024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5262024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-0" }
5272024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5282024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-M2", serial: "synthetic-serial-1" }
5292024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5302024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-0" }
5312024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5322024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-1" }
5332024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5342024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-2" }
5352024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5362024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-3" }
5372024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5382024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-4" }
5392024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5402024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-5" }
5412024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5422024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-6" }
5432024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5442024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-7" }
5452024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Removing disk file = sled-storage/src/resources.rs:532 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5462024-06-21T21:37:35.940ZINFOSledAgent (StorageResources): Not removing synthetic disk file = sled-storage/src/resources.rs:553 identity = DiskIdentity { vendor: "synthetic-vendor", model: "synthetic-model-U2", serial: "synthetic-serial-8" }
5472024-06-21T21:37:37.180ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_internal_dns_bd6e5965-968d-4c3d-aa50-17b3d5747fdb
5482024-06-21T21:37:37.180ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_internal_dns_33ad5a14-40c4-4b62-8de3-c1143a9861de
5492024-06-21T21:37:37.180ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_internal_dns_6c75629f-c9c3-4cd0-8ac6-814984e78c82
5502024-06-21T21:37:37.180ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5512024-06-21T21:37:37.180ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5522024-06-21T21:37:37.180ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 9991797 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:49431 req_id = 0100256f-8563-45ca-9bc4-1d91a9c62857 response_code = 204 uri = /omicron-zones
5532024-06-21T21:37:37.182ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:1::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5542024-06-21T21:37:37.258ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:2::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5552024-06-21T21:37:37.334ZINFOSledAgent (RSS): Configuring DNS server dns_config_addr = [fd00:1122:3344:3::1]:5353 file = sled-agent/src/rack_setup/service.rs:549
5562024-06-21T21:37:37.411ZINFOSledAgent (RSS): Configured all DNS servers file = sled-agent/src/rack_setup/service.rs:585
5572024-06-21T21:37:37.411ZINFOSledAgent (RSS): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
5582024-06-21T21:37:37.442ZINFOSledAgent (RSS): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
5592024-06-21T21:37:37.443ZINFOSledAgent (RSS): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
5602024-06-21T21:37:37.443ZINFOSledAgent (RSS): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
5612024-06-21T21:37:37.471ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(3), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
5622024-06-21T21:37:37.471ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:35300
5632024-06-21T21:37:37.471ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5642024-06-21T21:37:37.471ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
5652024-06-21T21:37:37.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5662024-06-21T21:37:37.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5672024-06-21T21:37:37.510ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
5682024-06-21T21:37:37.510ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
5692024-06-21T21:37:37.528ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5702024-06-21T21:37:37.528ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
5712024-06-21T21:37:37.528ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
5722024-06-21T21:37:37.529ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5732024-06-21T21:37:37.540ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
5742024-06-21T21:37:37.573ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
5752024-06-21T21:37:37.574ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
5762024-06-21T21:37:37.574ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
5772024-06-21T21:37:37.617ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte0", ip: 172.30.3.5, mac: MacAddr6([168, 64, 37, 255, 205, 184]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.3.1 }, vnic: "vopte0" } }
5782024-06-21T21:37:37.617ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1190 zone_type = ntp
5792024-06-21T21:37:37.618ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
5802024-06-21T21:37:37.618ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
5812024-06-21T21:37:37.618ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.169, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
5822024-06-21T21:37:37.619ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
5832024-06-21T21:37:37.641ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8 file = illumos-utils/src/zone.rs:315
5842024-06-21T21:37:37.660ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8 file = illumos-utils/src/zone.rs:354
5852024-06-21T21:37:38.302ZINFOSledAgent (ServiceManager): Profile for oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService4"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/chrony-setup"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="allow" value="fd00:1122:3344:100::/56"/> <propval type="boolean" name="boundary" value="true"/> <propval type="astring" name="server" value="0.pool.ntp.org"/> </property_group> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="1.1.1.1" /> <value_node value="9.9.9.9" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/ntp"> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte0"/> <propval type="astring" name="gateway" value="172.30.3.1"/> <propval type="astring" name="ip" value="172.30.3.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
5862024-06-21T21:37:38.303ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5872024-06-21T21:37:38.436ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 57.576188ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5882024-06-21T21:37:38.508ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 119.622093ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5892024-06-21T21:37:38.659ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 271.144416ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5902024-06-21T21:37:38.940ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 509.637736ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5912024-06-21T21:37:39.283ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45933
5922024-06-21T21:37:39.283ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 137 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45933 req_id = 376f57bd-a764-4881-8255-288226cf17bc response_code = 200 uri = /baseboard
5932024-06-21T21:37:39.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 1.161952438s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5942024-06-21T21:37:40.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 1.016057719s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5952024-06-21T21:37:41.656ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 682.153416ms file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5962024-06-21T21:37:42.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8") failed: Property not found. retry in 1.468580741s file = illumos-utils/src/svc.rs:36 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5972024-06-21T21:37:43.834ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8
5982024-06-21T21:37:43.834ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
5992024-06-21T21:37:43.835ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
6002024-06-21T21:37:43.835ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 6364128 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:35300 req_id = 65c95a9f-35ce-408a-aa96-acb14e14b5f1 response_code = 204 uri = /omicron-zones
6012024-06-21T21:37:43.837ZINFOSledAgent (RSS): Waiting for rack time synchronization file = sled-agent/src/rack_setup/service.rs:626
6022024-06-21T21:37:43.866ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6032024-06-21T21:37:43.866ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40471
6042024-06-21T21:37:43.876ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3533
6052024-06-21T21:37:43.876ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 9645 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40471 req_id = 873dd655-addd-4279-a0a7-8a3e958a878c response_code = 500 uri = /timesync
6062024-06-21T21:37:43.878ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6072024-06-21T21:37:44.039ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6082024-06-21T21:37:44.040ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:60689
6092024-06-21T21:37:48.048ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3533
6102024-06-21T21:37:48.049ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 4008892 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:60689 req_id = 571f703f-3393-4152-9a03-be9855571b77 response_code = 500 uri = /timesync
6112024-06-21T21:37:48.051ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6122024-06-21T21:37:48.183ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6132024-06-21T21:37:48.184ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:49681
6142024-06-21T21:37:48.193ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3533
6152024-06-21T21:37:48.194ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 9845 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:49681 req_id = 4d97878a-b6f1-4d51-8fc3-bd9142a1b65b response_code = 500 uri = /timesync
6162024-06-21T21:37:48.197ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6172024-06-21T21:37:48.441ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6182024-06-21T21:37:48.441ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:32975
6192024-06-21T21:37:48.449ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3533
6202024-06-21T21:37:48.450ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 8496 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:32975 req_id = 0b24dfaf-ea38-4bf5-9d29-196c1b7755bd response_code = 500 uri = /timesync
6212024-06-21T21:37:48.452ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6222024-06-21T21:37:48.729ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6232024-06-21T21:37:48.729ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:40819
6242024-06-21T21:37:49.737ZINFOSledAgent (ServiceManager): chronyc command failed: Error running command in zone 'oxz_ntp_0516b982-c4cb-4196-b8d1-f13d36b135a8': Command [/usr/bin/chronyc -c tracking] executed and failed with status: exit status: 1 stdout: 506 Cannot talk to daemon stderr: file = sled-agent/src/services.rs:3533
6252024-06-21T21:37:49.737ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = NTP zone not ready file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 1008678 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:40819 req_id = 39a4712e-cec1-4f5d-a472-92962f1c4ba4 response_code = 500 uri = /timesync
6262024-06-21T21:37:49.742ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6272024-06-21T21:37:50.055ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6282024-06-21T21:37:50.055ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50251
6292024-06-21T21:37:50.064ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8958 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50251 req_id = 1d07f28b-f622-43a3-9849-cd79119e14a1 response_code = 200 uri = /timesync
6302024-06-21T21:37:50.068ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6312024-06-21T21:37:50.068ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6322024-06-21T21:37:50.339ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6332024-06-21T21:37:50.339ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48671
6342024-06-21T21:37:50.348ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:48671 req_id = e85254f1-a74e-4906-be58-07e219acb5c6 response_code = 200 uri = /timesync
6352024-06-21T21:37:50.351ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6362024-06-21T21:37:50.351ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6372024-06-21T21:37:50.607ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6382024-06-21T21:37:50.607ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:64482
6392024-06-21T21:37:50.615ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8379 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:64482 req_id = 22b3eb67-d503-4c5c-bcba-bf6b62bac69c response_code = 200 uri = /timesync
6402024-06-21T21:37:50.617ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6412024-06-21T21:37:50.617ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6422024-06-21T21:37:50.902ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6432024-06-21T21:37:50.902ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47820
6442024-06-21T21:37:50.911ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8492 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47820 req_id = 624e57b6-14ff-4eff-89bf-a045560ca3d7 response_code = 200 uri = /timesync
6452024-06-21T21:37:50.912ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6462024-06-21T21:37:50.912ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6472024-06-21T21:37:51.207ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6482024-06-21T21:37:51.207ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:52715
6492024-06-21T21:37:51.216ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8866 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:52715 req_id = d3ae31a4-4d7d-46a0-8e8d-0e0965bc1783 response_code = 200 uri = /timesync
6502024-06-21T21:37:51.219ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6512024-06-21T21:37:51.219ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6522024-06-21T21:37:51.676ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6532024-06-21T21:37:51.676ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:44160
6542024-06-21T21:37:51.684ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7737 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:44160 req_id = 44364170-0627-45c1-a5dd-c7e8af12c4a5 response_code = 200 uri = /timesync
6552024-06-21T21:37:51.686ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6562024-06-21T21:37:51.686ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6572024-06-21T21:37:52.252ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6582024-06-21T21:37:52.252ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33308
6592024-06-21T21:37:52.260ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7931 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:33308 req_id = 53f79b90-006c-41ad-ab79-d029554dff36 response_code = 200 uri = /timesync
6602024-06-21T21:37:52.263ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6612024-06-21T21:37:52.263ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6622024-06-21T21:37:53.228ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 363.805132ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
6632024-06-21T21:37:53.374ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6642024-06-21T21:37:53.374ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:47145
6652024-06-21T21:37:53.382ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 7951 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:47145 req_id = 192a57a1-3090-4466-8f48-0ac0acfa9107 response_code = 200 uri = /timesync
6662024-06-21T21:37:53.385ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6672024-06-21T21:37:53.385ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6682024-06-21T21:37:54.408ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6692024-06-21T21:37:54.408ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:50015
6702024-06-21T21:37:54.416ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 8437 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:50015 req_id = 7244b353-b6ff-46d9-a3ec-a2664f9d8b20 response_code = 200 uri = /timesync
6712024-06-21T21:37:54.420ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 0, ip_addr: ::, stratum: 0, ref_time: 0.0, correction: 0.0 } file = sled-agent/src/rack_setup/service.rs:634
6722024-06-21T21:37:54.420ZWARNSledAgent (RSS): Time is not yet synchronized error = "Time is synchronized on 0/1 sleds" file = sled-agent/src/rack_setup/service.rs:656
6732024-06-21T21:37:55.273ZINFOSledAgent (RSS): Checking time synchronization for [fd00:1122:3344:101::1]:12345... file = sled-agent/src/rack_setup/service.rs:606
6742024-06-21T21:37:55.273ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:63814
6752024-06-21T21:37:55.293ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 20178 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::1]:63814 req_id = e9c80faf-9b38-49ca-aa6a-13e334c91b5d response_code = 200 uri = /timesync
6762024-06-21T21:37:55.297ZINFOSledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: true, ref_id: 2728380539, ip_addr: 162.159.200.123, stratum: 4, ref_time: 1719005874.239101, correction: -0.0 } file = sled-agent/src/rack_setup/service.rs:634
6772024-06-21T21:37:55.297ZINFOSledAgent (RSS): Finished setting up Internal DNS and NTP file = sled-agent/src/rack_setup/service.rs:1224
6782024-06-21T21:37:55.326ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(4), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
6792024-06-21T21:37:55.326ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:57987
6802024-06-21T21:37:55.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6812024-06-21T21:37:55.327ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
6822024-06-21T21:37:55.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6832024-06-21T21:37:55.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6842024-06-21T21:37:55.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
6852024-06-21T21:37:55.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
6862024-06-21T21:37:55.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6872024-06-21T21:37:55.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6882024-06-21T21:37:55.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6892024-06-21T21:37:55.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6902024-06-21T21:37:55.546ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6912024-06-21T21:37:55.546ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6922024-06-21T21:37:55.632ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6932024-06-21T21:37:55.632ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6942024-06-21T21:37:55.713ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
6952024-06-21T21:37:55.713ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
6962024-06-21T21:37:55.797ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
6972024-06-21T21:37:55.797ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
6982024-06-21T21:37:55.797ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
6992024-06-21T21:37:55.797ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7002024-06-21T21:37:55.883ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7012024-06-21T21:37:55.883ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7022024-06-21T21:37:55.883ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7032024-06-21T21:37:55.883ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7042024-06-21T21:37:55.883ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
7052024-06-21T21:37:55.975ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d file = illumos-utils/src/zone.rs:315
7062024-06-21T21:37:55.979ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348 file = illumos-utils/src/zone.rs:315
7072024-06-21T21:37:55.983ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2 file = illumos-utils/src/zone.rs:315
7082024-06-21T21:37:55.988ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc file = illumos-utils/src/zone.rs:315
7092024-06-21T21:37:55.993ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9 file = illumos-utils/src/zone.rs:315
7102024-06-21T21:37:55.997ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d file = illumos-utils/src/zone.rs:354
7112024-06-21T21:37:56.002ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348 file = illumos-utils/src/zone.rs:354
7122024-06-21T21:37:56.007ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2 file = illumos-utils/src/zone.rs:354
7132024-06-21T21:37:56.012ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc file = illumos-utils/src/zone.rs:354
7142024-06-21T21:37:56.016ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9 file = illumos-utils/src/zone.rs:354
7152024-06-21T21:37:58.972ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService5"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::3"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="bd58e908-3308-4fab-8db4-ed67150a5a5d"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::3]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::3]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7162024-06-21T21:37:58.972ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7172024-06-21T21:37:58.978ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService9"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="9a9094ea-c1a1-4c5e-be6a-e79e1be407c9"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::5]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::5]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7182024-06-21T21:37:58.978ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7192024-06-21T21:37:59.019ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService6"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="c999f854-acb5-47bf-95d4-4879391f2348"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::6]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::6]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7202024-06-21T21:37:59.019ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7212024-06-21T21:37:59.080ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService8"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="0aa1beae-e845-4690-9fd2-bf35912e37fc"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::7]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::7]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7222024-06-21T21:37:59.080ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7232024-06-21T21:37:59.088ZINFOSledAgent (ServiceManager): Profile for oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService7"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::4"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/cockroachdb"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="oxide/cockroach-admin"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="zone_id" value="6ec51f86-b813-4d2c-bee0-1400d33a00d2"/> <propval type="astring" name="cockroach_address" value="[fd00:1122:3344:101::4]:32221"/> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::4]:32222"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
7242024-06-21T21:37:59.088ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7252024-06-21T21:37:59.126ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 70.492659ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7262024-06-21T21:37:59.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 74.382ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7272024-06-21T21:37:59.163ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 57.108415ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7282024-06-21T21:37:59.301ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 67.471843ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7292024-06-21T21:37:59.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 111.004494ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7302024-06-21T21:37:59.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 85.120564ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7312024-06-21T21:37:59.899ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 74.555535ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7322024-06-21T21:38:00.060ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 106.377753ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7332024-06-21T21:38:00.140ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 295.762935ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7342024-06-21T21:38:00.403ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 297.323673ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7352024-06-21T21:38:00.509ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 53.651114ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7362024-06-21T21:38:00.592ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 122.759142ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7372024-06-21T21:38:00.640ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 279.871481ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7382024-06-21T21:38:00.684ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 428.726711ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7392024-06-21T21:38:00.727ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 82.303575ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7402024-06-21T21:38:00.765ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 253.349786ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7412024-06-21T21:38:00.846ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 300.100452ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7422024-06-21T21:38:00.946ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 196.822472ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7432024-06-21T21:38:01.062ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.039767474s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7442024-06-21T21:38:01.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 530.693888ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7452024-06-21T21:38:01.348ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 902.665245ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7462024-06-21T21:38:01.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 720.167613ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7472024-06-21T21:38:01.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 225.389355ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7482024-06-21T21:38:01.665ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 768.704527ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7492024-06-21T21:38:01.686ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.094409728s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7502024-06-21T21:38:02.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 668.02131ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7512024-06-21T21:38:02.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.458850604s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7522024-06-21T21:38:02.262ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.443079895s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7532024-06-21T21:38:02.445ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.047059412s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7542024-06-21T21:38:02.789ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.339044907s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7552024-06-21T21:38:02.797ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 538.642175ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7562024-06-21T21:38:03.345ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.490357758s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7572024-06-21T21:38:03.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 563.639769ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7582024-06-21T21:38:03.589ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.388983977s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7592024-06-21T21:38:03.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 898.342836ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7602024-06-21T21:38:04.075ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 597.990565ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7612024-06-21T21:38:04.138ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 581.873932ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7622024-06-21T21:38:04.621ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 977.898724ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7632024-06-21T21:38:04.681ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 787.59321ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7642024-06-21T21:38:04.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 623.748352ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7652024-06-21T21:38:04.845ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 538.717411ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7662024-06-21T21:38:04.987ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.42500721s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7672024-06-21T21:38:05.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.149332265s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7682024-06-21T21:38:05.391ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.243839422s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7692024-06-21T21:38:05.477ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.360745273s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7702024-06-21T21:38:05.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 843.02537ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7712024-06-21T21:38:06.422ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 682.166379ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7722024-06-21T21:38:06.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 744.936801ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7732024-06-21T21:38:06.518ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 531.526134ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7742024-06-21T21:38:06.643ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 815.333021ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7752024-06-21T21:38:06.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.400808732s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7762024-06-21T21:38:07.057ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 726.949156ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7772024-06-21T21:38:07.113ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.424329527s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7782024-06-21T21:38:07.210ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.154469781s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7792024-06-21T21:38:07.466ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.275509656s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7802024-06-21T21:38:07.792ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 886.863378ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7812024-06-21T21:38:08.257ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.189769801s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7822024-06-21T21:38:08.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.48750843s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7832024-06-21T21:38:08.545ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 930.701284ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7842024-06-21T21:38:08.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.295612709s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7852024-06-21T21:38:08.750ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.358146998s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7862024-06-21T21:38:08.866ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 748.470692ms error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
7872024-06-21T21:38:09.457ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.097142601s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7882024-06-21T21:38:09.485ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 830.527327ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7892024-06-21T21:38:09.558ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36324
7902024-06-21T21:38:09.558ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 100 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36324 req_id = 4a2cbd6e-141a-46c8-9f6c-eddd6f0a27f0 response_code = 200 uri = /baseboard
7912024-06-21T21:38:09.870ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.132602858s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7922024-06-21T21:38:09.992ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.119747395s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7932024-06-21T21:38:10.117ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 624.787598ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7942024-06-21T21:38:10.325ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 931.120645ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
7952024-06-21T21:38:10.562ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 796.486831ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
7962024-06-21T21:38:10.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.321874414s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
7972024-06-21T21:38:11.013ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.324933773s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
7982024-06-21T21:38:11.121ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 839.795027ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
7992024-06-21T21:38:11.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 545.406511ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8002024-06-21T21:38:11.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 788.748911ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8012024-06-21T21:38:11.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 694.66645ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8022024-06-21T21:38:11.969ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 718.437466ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8032024-06-21T21:38:12.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 638.432123ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8042024-06-21T21:38:12.166ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 863.367372ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8052024-06-21T21:38:12.347ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 591.726736ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8062024-06-21T21:38:12.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.313946282s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8072024-06-21T21:38:12.697ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.428312622s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8082024-06-21T21:38:12.735ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.494448565s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8092024-06-21T21:38:12.949ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.276644931s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8102024-06-21T21:38:13.040ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.05798296s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8112024-06-21T21:38:13.847ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 577.338846ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8122024-06-21T21:38:14.109ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 887.241308ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8132024-06-21T21:38:14.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 563.469405ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8142024-06-21T21:38:14.235ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.038143518s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8152024-06-21T21:38:14.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.318692239s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8162024-06-21T21:38:14.434ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.123183952s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8172024-06-21T21:38:14.710ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 507.392276ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8182024-06-21T21:38:15.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.226824692s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8192024-06-21T21:38:15.227ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.491527046s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8202024-06-21T21:38:15.283ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 769.271375ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8212024-06-21T21:38:15.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 1.15982121s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8222024-06-21T21:38:15.574ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 797.510496ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8232024-06-21T21:38:16.061ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 584.122179ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8242024-06-21T21:38:16.240ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 530.578279ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8252024-06-21T21:38:16.381ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.428979638s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8262024-06-21T21:38:16.654ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 625.568213ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8272024-06-21T21:38:16.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.260664029s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8282024-06-21T21:38:16.736ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 801.370531ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8292024-06-21T21:38:16.779ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 761.144978ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8302024-06-21T21:38:17.290ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d") failed: Property not found. retry in 1.483480508s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8312024-06-21T21:38:17.547ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 592.364189ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8322024-06-21T21:38:17.554ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2") failed: Property not found. retry in 1.260816843s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8332024-06-21T21:38:17.819ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc") failed: Property not found. retry in 1.128407061s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8342024-06-21T21:38:17.998ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.249531461s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8352024-06-21T21:38:18.149ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9") failed: Property not found. retry in 652.747636ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8362024-06-21T21:38:19.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.223597188s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8372024-06-21T21:38:20.491ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.260293023s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8382024-06-21T21:38:21.762ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 528.722979ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8392024-06-21T21:38:22.311ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.026926814s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8402024-06-21T21:38:23.349ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 998.177981ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8412024-06-21T21:38:24.368ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.320385875s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8422024-06-21T21:38:24.653ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.435655619s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
8432024-06-21T21:38:25.699ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 1.308398959s file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8442024-06-21T21:38:27.018ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 562.03608ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8452024-06-21T21:38:27.593ZWARNSledAgent (ServiceManager): wait for service Some("oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348") failed: Property not found. retry in 631.549951ms file = illumos-utils/src/svc.rs:36 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8462024-06-21T21:38:28.246ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_cockroachdb_bd58e908-3308-4fab-8db4-ed67150a5a5d
8472024-06-21T21:38:28.246ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_cockroachdb_c999f854-acb5-47bf-95d4-4879391f2348
8482024-06-21T21:38:28.246ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_cockroachdb_6ec51f86-b813-4d2c-bee0-1400d33a00d2
8492024-06-21T21:38:28.246ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_cockroachdb_0aa1beae-e845-4690-9fd2-bf35912e37fc
8502024-06-21T21:38:28.246ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_cockroachdb_9a9094ea-c1a1-4c5e-be6a-e79e1be407c9
8512024-06-21T21:38:28.246ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8522024-06-21T21:38:28.247ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
8532024-06-21T21:38:28.248ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 32944936 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:57987 req_id = 394fae0b-2fbd-433d-beb2-11256193b8da response_code = 204 uri = /omicron-zones
8542024-06-21T21:38:28.279ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:48015
8552024-06-21T21:38:28.308ZINFOSledAgent (ServiceManager): Initializing CRDB Cluster - sending request to [fd00:1122:3344:101::7]:32221 file = sled-agent/src/services.rs:3367
8562024-06-21T21:38:28.477ZINFOSledAgent (ServiceManager): Formatting CRDB file = sled-agent/src/services.rs:3385
8572024-06-21T21:38:31.893ZINFOSledAgent (ServiceManager): Formatting CRDB - Completed file = sled-agent/src/services.rs:3408
8582024-06-21T21:38:31.893ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3613779 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::1]:48015 req_id = e5851eb8-592c-4a59-88a1-8212bd3c38bc response_code = 204 uri = /cockroachdb
8592024-06-21T21:38:31.925ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
8602024-06-21T21:38:31.925ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::1]:33425
8612024-06-21T21:38:31.926ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8622024-06-21T21:38:31.926ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
8632024-06-21T21:38:31.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8642024-06-21T21:38:31.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8652024-06-21T21:38:31.969ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8662024-06-21T21:38:31.969ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
8672024-06-21T21:38:31.988ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8682024-06-21T21:38:31.988ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8692024-06-21T21:38:32.006ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8702024-06-21T21:38:32.006ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8712024-06-21T21:38:32.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8722024-06-21T21:38:32.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8732024-06-21T21:38:32.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8742024-06-21T21:38:32.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8752024-06-21T21:38:32.060ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
8762024-06-21T21:38:32.060ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
8772024-06-21T21:38:32.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8782024-06-21T21:38:32.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8792024-06-21T21:38:32.142ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
8802024-06-21T21:38:32.142ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
8812024-06-21T21:38:32.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
8822024-06-21T21:38:32.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
8832024-06-21T21:38:32.288ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8842024-06-21T21:38:32.288ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
8852024-06-21T21:38:32.359ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8862024-06-21T21:38:32.359ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
8872024-06-21T21:38:32.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8882024-06-21T21:38:32.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
8892024-06-21T21:38:32.507ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8902024-06-21T21:38:32.507ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
8912024-06-21T21:38:32.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8922024-06-21T21:38:32.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
8932024-06-21T21:38:32.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8942024-06-21T21:38:32.648ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
8952024-06-21T21:38:32.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8962024-06-21T21:38:32.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
8972024-06-21T21:38:32.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
8982024-06-21T21:38:32.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
8992024-06-21T21:38:32.838ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9002024-06-21T21:38:32.838ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
9012024-06-21T21:38:32.897ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9022024-06-21T21:38:32.897ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
9032024-06-21T21:38:32.898ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
9042024-06-21T21:38:32.898ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9052024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
9062024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
9072024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
9082024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
9092024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
9102024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
9112024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
9122024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
9132024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
9142024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
9152024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
9162024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
9172024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
9182024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
9192024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
9202024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
9212024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
9222024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
9232024-06-21T21:38:32.993ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
9242024-06-21T21:38:32.994ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 1068037 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 5f8fcb66-4104-4ff4-a3c0-bd9c093bd934 response_code = 500 uri = /omicron-zones
9252024-06-21T21:38:32.994ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "5f8fcb66-4104-4ff4-a3c0-bd9c093bd934", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "5f8fcb66-4104-4ff4-a3c0-bd9c093bd934" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 54.531791ms sled_address = [fd00:1122:3344:101::1]:12345
9262024-06-21T21:38:33.050ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
9272024-06-21T21:38:33.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9282024-06-21T21:38:33.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
9292024-06-21T21:38:33.073ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9302024-06-21T21:38:33.073ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9312024-06-21T21:38:33.092ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9322024-06-21T21:38:33.092ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9332024-06-21T21:38:33.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9342024-06-21T21:38:33.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9352024-06-21T21:38:33.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9362024-06-21T21:38:33.129ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9372024-06-21T21:38:33.147ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9382024-06-21T21:38:33.147ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9392024-06-21T21:38:33.165ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9402024-06-21T21:38:33.165ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9412024-06-21T21:38:33.183ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
9422024-06-21T21:38:33.183ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
9432024-06-21T21:38:33.204ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9442024-06-21T21:38:33.204ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9452024-06-21T21:38:33.224ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9462024-06-21T21:38:33.224ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
9472024-06-21T21:38:33.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
9482024-06-21T21:38:33.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
9492024-06-21T21:38:33.261ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9502024-06-21T21:38:33.261ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
9512024-06-21T21:38:33.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9522024-06-21T21:38:33.279ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
9532024-06-21T21:38:33.297ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9542024-06-21T21:38:33.297ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
9552024-06-21T21:38:33.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9562024-06-21T21:38:33.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
9572024-06-21T21:38:33.331ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9582024-06-21T21:38:33.331ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
9592024-06-21T21:38:33.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9602024-06-21T21:38:33.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
9612024-06-21T21:38:33.365ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9622024-06-21T21:38:33.365ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
9632024-06-21T21:38:33.385ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9642024-06-21T21:38:33.385ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
9652024-06-21T21:38:33.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
9662024-06-21T21:38:33.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
9672024-06-21T21:38:33.424ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9682024-06-21T21:38:33.424ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
9692024-06-21T21:38:33.425ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
9702024-06-21T21:38:33.425ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
9712024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
9722024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
9732024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
9742024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
9752024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
9762024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
9772024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
9782024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
9792024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
9802024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
9812024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
9822024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
9832024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
9842024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
9852024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
9862024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
9872024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
9882024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
9892024-06-21T21:38:33.523ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
9902024-06-21T21:38:33.523ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 473107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 4a9ddb30-fd55-4f97-be97-575f9df0ca47 response_code = 500 uri = /omicron-zones
9912024-06-21T21:38:33.524ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4a9ddb30-fd55-4f97-be97-575f9df0ca47", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:32 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4a9ddb30-fd55-4f97-be97-575f9df0ca47" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 77.420509ms sled_address = [fd00:1122:3344:101::1]:12345
9922024-06-21T21:38:33.602ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
9932024-06-21T21:38:33.603ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9942024-06-21T21:38:33.603ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
9952024-06-21T21:38:33.625ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9962024-06-21T21:38:33.625ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9972024-06-21T21:38:33.647ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
9982024-06-21T21:38:33.647ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
9992024-06-21T21:38:33.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10002024-06-21T21:38:33.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10012024-06-21T21:38:33.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10022024-06-21T21:38:33.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10032024-06-21T21:38:33.709ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10042024-06-21T21:38:33.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10052024-06-21T21:38:33.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10062024-06-21T21:38:33.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10072024-06-21T21:38:33.748ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10082024-06-21T21:38:33.748ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10092024-06-21T21:38:33.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10102024-06-21T21:38:33.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10112024-06-21T21:38:33.791ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10122024-06-21T21:38:33.791ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10132024-06-21T21:38:33.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
10142024-06-21T21:38:33.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10152024-06-21T21:38:33.834ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10162024-06-21T21:38:33.834ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
10172024-06-21T21:38:33.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10182024-06-21T21:38:33.853ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
10192024-06-21T21:38:33.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10202024-06-21T21:38:33.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10212024-06-21T21:38:33.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10222024-06-21T21:38:33.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
10232024-06-21T21:38:33.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10242024-06-21T21:38:33.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
10252024-06-21T21:38:33.929ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10262024-06-21T21:38:33.929ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
10272024-06-21T21:38:33.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10282024-06-21T21:38:33.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
10292024-06-21T21:38:33.969ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10302024-06-21T21:38:33.969ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
10312024-06-21T21:38:33.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10322024-06-21T21:38:33.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
10332024-06-21T21:38:34.009ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10342024-06-21T21:38:34.009ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
10352024-06-21T21:38:34.010ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
10362024-06-21T21:38:34.010ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
10372024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
10382024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
10392024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
10402024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
10412024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
10422024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
10432024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
10442024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
10452024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
10462024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
10472024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
10482024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
10492024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
10502024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
10512024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
10522024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
10532024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
10542024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
10552024-06-21T21:38:34.107ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
10562024-06-21T21:38:34.107ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 505415 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = d0fc0449-0cb1-40da-8418-313594cf3dc2 response_code = 500 uri = /omicron-zones
10572024-06-21T21:38:34.107ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d0fc0449-0cb1-40da-8418-313594cf3dc2", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d0fc0449-0cb1-40da-8418-313594cf3dc2" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 101.979999ms sled_address = [fd00:1122:3344:101::1]:12345
10582024-06-21T21:38:34.211ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
10592024-06-21T21:38:34.211ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10602024-06-21T21:38:34.211ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
10612024-06-21T21:38:34.233ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10622024-06-21T21:38:34.233ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10632024-06-21T21:38:34.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10642024-06-21T21:38:34.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
10652024-06-21T21:38:34.276ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10662024-06-21T21:38:34.276ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10672024-06-21T21:38:34.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10682024-06-21T21:38:34.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10692024-06-21T21:38:34.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10702024-06-21T21:38:34.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10712024-06-21T21:38:34.335ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10722024-06-21T21:38:34.335ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10732024-06-21T21:38:34.358ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
10742024-06-21T21:38:34.358ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
10752024-06-21T21:38:34.379ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10762024-06-21T21:38:34.379ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10772024-06-21T21:38:34.399ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
10782024-06-21T21:38:34.399ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
10792024-06-21T21:38:34.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
10802024-06-21T21:38:34.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
10812024-06-21T21:38:34.440ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10822024-06-21T21:38:34.440ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
10832024-06-21T21:38:34.459ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10842024-06-21T21:38:34.459ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
10852024-06-21T21:38:34.476ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10862024-06-21T21:38:34.476ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
10872024-06-21T21:38:34.496ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10882024-06-21T21:38:34.496ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
10892024-06-21T21:38:34.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10902024-06-21T21:38:34.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
10912024-06-21T21:38:34.531ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10922024-06-21T21:38:34.531ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
10932024-06-21T21:38:34.552ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10942024-06-21T21:38:34.552ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
10952024-06-21T21:38:34.573ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10962024-06-21T21:38:34.573ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
10972024-06-21T21:38:34.592ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
10982024-06-21T21:38:34.592ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
10992024-06-21T21:38:34.612ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11002024-06-21T21:38:34.612ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
11012024-06-21T21:38:34.612ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
11022024-06-21T21:38:34.612ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11032024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
11042024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
11052024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
11062024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
11072024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
11082024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
11092024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
11102024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
11112024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
11122024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
11132024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
11142024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
11152024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
11162024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
11172024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
11182024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
11192024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
11202024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
11212024-06-21T21:38:34.706ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
11222024-06-21T21:38:34.706ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 495249 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 63a79cd1-5400-44fe-998c-727fd3773451 response_code = 500 uri = /omicron-zones
11232024-06-21T21:38:34.706ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "63a79cd1-5400-44fe-998c-727fd3773451", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:34 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "63a79cd1-5400-44fe-998c-727fd3773451" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 180.514598ms sled_address = [fd00:1122:3344:101::1]:12345
11242024-06-21T21:38:34.888ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
11252024-06-21T21:38:34.888ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11262024-06-21T21:38:34.888ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
11272024-06-21T21:38:34.913ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11282024-06-21T21:38:34.913ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11292024-06-21T21:38:34.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11302024-06-21T21:38:34.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11312024-06-21T21:38:34.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11322024-06-21T21:38:34.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11332024-06-21T21:38:34.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11342024-06-21T21:38:34.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11352024-06-21T21:38:34.996ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11362024-06-21T21:38:34.996ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11372024-06-21T21:38:35.017ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11382024-06-21T21:38:35.017ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11392024-06-21T21:38:35.038ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11402024-06-21T21:38:35.038ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11412024-06-21T21:38:35.059ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11422024-06-21T21:38:35.059ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11432024-06-21T21:38:35.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11442024-06-21T21:38:35.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
11452024-06-21T21:38:35.101ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
11462024-06-21T21:38:35.101ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
11472024-06-21T21:38:35.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11482024-06-21T21:38:35.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
11492024-06-21T21:38:35.141ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11502024-06-21T21:38:35.141ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
11512024-06-21T21:38:35.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11522024-06-21T21:38:35.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
11532024-06-21T21:38:35.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11542024-06-21T21:38:35.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
11552024-06-21T21:38:35.197ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11562024-06-21T21:38:35.197ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
11572024-06-21T21:38:35.218ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11582024-06-21T21:38:35.218ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
11592024-06-21T21:38:35.237ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11602024-06-21T21:38:35.237ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
11612024-06-21T21:38:35.257ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11622024-06-21T21:38:35.257ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
11632024-06-21T21:38:35.277ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
11642024-06-21T21:38:35.277ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
11652024-06-21T21:38:35.295ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11662024-06-21T21:38:35.295ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
11672024-06-21T21:38:35.296ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
11682024-06-21T21:38:35.296ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
11692024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
11702024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
11712024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
11722024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
11732024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
11742024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
11752024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
11762024-06-21T21:38:35.389ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
11772024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
11782024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
11792024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
11802024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
11812024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
11822024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
11832024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
11842024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
11852024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
11862024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
11872024-06-21T21:38:35.390ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
11882024-06-21T21:38:35.390ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 501828 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = ef573358-6fa5-42df-96af-158b71fa793a response_code = 500 uri = /omicron-zones
11892024-06-21T21:38:35.390ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "ef573358-6fa5-42df-96af-158b71fa793a", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "ef573358-6fa5-42df-96af-158b71fa793a" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 196.106866ms sled_address = [fd00:1122:3344:101::1]:12345
11902024-06-21T21:38:35.587ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
11912024-06-21T21:38:35.588ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11922024-06-21T21:38:35.588ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
11932024-06-21T21:38:35.610ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11942024-06-21T21:38:35.610ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11952024-06-21T21:38:35.631ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
11962024-06-21T21:38:35.631ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
11972024-06-21T21:38:35.651ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
11982024-06-21T21:38:35.651ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
11992024-06-21T21:38:35.670ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12002024-06-21T21:38:35.670ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12012024-06-21T21:38:35.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12022024-06-21T21:38:35.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12032024-06-21T21:38:35.711ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12042024-06-21T21:38:35.711ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12052024-06-21T21:38:35.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12062024-06-21T21:38:35.732ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12072024-06-21T21:38:35.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12082024-06-21T21:38:35.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12092024-06-21T21:38:35.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12102024-06-21T21:38:35.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12112024-06-21T21:38:35.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
12122024-06-21T21:38:35.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12132024-06-21T21:38:35.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12142024-06-21T21:38:35.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
12152024-06-21T21:38:35.839ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12162024-06-21T21:38:35.839ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
12172024-06-21T21:38:35.860ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12182024-06-21T21:38:35.860ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12192024-06-21T21:38:35.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12202024-06-21T21:38:35.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
12212024-06-21T21:38:35.900ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12222024-06-21T21:38:35.900ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
12232024-06-21T21:38:35.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12242024-06-21T21:38:35.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
12252024-06-21T21:38:35.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12262024-06-21T21:38:35.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
12272024-06-21T21:38:35.961ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12282024-06-21T21:38:35.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
12292024-06-21T21:38:35.982ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12302024-06-21T21:38:35.983ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
12312024-06-21T21:38:36.002ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12322024-06-21T21:38:36.002ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
12332024-06-21T21:38:36.002ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
12342024-06-21T21:38:36.003ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12352024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
12362024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
12372024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
12382024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
12392024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
12402024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
12412024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
12422024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
12432024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
12442024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
12452024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
12462024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
12472024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
12482024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
12492024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
12502024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
12512024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
12522024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
12532024-06-21T21:38:36.110ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
12542024-06-21T21:38:36.110ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 522735 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 8e14fcad-ae3f-401a-8a17-250412a17dbb response_code = 500 uri = /omicron-zones
12552024-06-21T21:38:36.110ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8e14fcad-ae3f-401a-8a17-250412a17dbb", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8e14fcad-ae3f-401a-8a17-250412a17dbb" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 365.885142ms sled_address = [fd00:1122:3344:101::1]:12345
12562024-06-21T21:38:36.478ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
12572024-06-21T21:38:36.478ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12582024-06-21T21:38:36.478ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
12592024-06-21T21:38:36.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12602024-06-21T21:38:36.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12612024-06-21T21:38:36.523ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12622024-06-21T21:38:36.523ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
12632024-06-21T21:38:36.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12642024-06-21T21:38:36.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12652024-06-21T21:38:36.564ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12662024-06-21T21:38:36.564ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12672024-06-21T21:38:36.585ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12682024-06-21T21:38:36.585ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12692024-06-21T21:38:36.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12702024-06-21T21:38:36.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12712024-06-21T21:38:36.624ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
12722024-06-21T21:38:36.624ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
12732024-06-21T21:38:36.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12742024-06-21T21:38:36.645ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12752024-06-21T21:38:36.665ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
12762024-06-21T21:38:36.665ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
12772024-06-21T21:38:36.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
12782024-06-21T21:38:36.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
12792024-06-21T21:38:36.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12802024-06-21T21:38:36.707ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
12812024-06-21T21:38:36.727ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12822024-06-21T21:38:36.727ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
12832024-06-21T21:38:36.749ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12842024-06-21T21:38:36.749ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
12852024-06-21T21:38:36.768ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12862024-06-21T21:38:36.768ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
12872024-06-21T21:38:36.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12882024-06-21T21:38:36.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
12892024-06-21T21:38:36.807ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12902024-06-21T21:38:36.807ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
12912024-06-21T21:38:36.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12922024-06-21T21:38:36.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
12932024-06-21T21:38:36.845ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12942024-06-21T21:38:36.845ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
12952024-06-21T21:38:36.865ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
12962024-06-21T21:38:36.865ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
12972024-06-21T21:38:36.886ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
12982024-06-21T21:38:36.886ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
12992024-06-21T21:38:36.886ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
13002024-06-21T21:38:36.886ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13012024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
13022024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
13032024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
13042024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
13052024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
13062024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
13072024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
13082024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
13092024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
13102024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
13112024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
13122024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
13132024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
13142024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
13152024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
13162024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
13172024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
13182024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
13192024-06-21T21:38:36.983ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
13202024-06-21T21:38:36.983ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 505391 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 2871ff30-1574-4657-a70b-1499c32b1d8d response_code = 500 uri = /omicron-zones
13212024-06-21T21:38:36.984ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2871ff30-1574-4657-a70b-1499c32b1d8d", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:36 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2871ff30-1574-4657-a70b-1499c32b1d8d" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 184.922039ms sled_address = [fd00:1122:3344:101::1]:12345
13222024-06-21T21:38:37.170ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
13232024-06-21T21:38:37.170ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13242024-06-21T21:38:37.170ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
13252024-06-21T21:38:37.192ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13262024-06-21T21:38:37.192ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13272024-06-21T21:38:37.214ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13282024-06-21T21:38:37.214ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13292024-06-21T21:38:37.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13302024-06-21T21:38:37.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13312024-06-21T21:38:37.253ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13322024-06-21T21:38:37.253ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13332024-06-21T21:38:37.272ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13342024-06-21T21:38:37.272ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13352024-06-21T21:38:37.291ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13362024-06-21T21:38:37.291ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13372024-06-21T21:38:37.310ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13382024-06-21T21:38:37.310ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13392024-06-21T21:38:37.330ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13402024-06-21T21:38:37.330ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13412024-06-21T21:38:37.350ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13422024-06-21T21:38:37.350ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
13432024-06-21T21:38:37.370ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
13442024-06-21T21:38:37.370ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
13452024-06-21T21:38:37.390ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13462024-06-21T21:38:37.390ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
13472024-06-21T21:38:37.410ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13482024-06-21T21:38:37.410ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
13492024-06-21T21:38:37.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13502024-06-21T21:38:37.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
13512024-06-21T21:38:37.452ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13522024-06-21T21:38:37.452ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
13532024-06-21T21:38:37.472ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13542024-06-21T21:38:37.472ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
13552024-06-21T21:38:37.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13562024-06-21T21:38:37.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
13572024-06-21T21:38:37.516ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13582024-06-21T21:38:37.516ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
13592024-06-21T21:38:37.535ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13602024-06-21T21:38:37.535ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
13612024-06-21T21:38:37.556ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
13622024-06-21T21:38:37.556ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
13632024-06-21T21:38:37.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13642024-06-21T21:38:37.578ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
13652024-06-21T21:38:37.578ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
13662024-06-21T21:38:37.578ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
13672024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
13682024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
13692024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
13702024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
13712024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
13722024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
13732024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
13742024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
13752024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
13762024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
13772024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
13782024-06-21T21:38:37.674ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
13792024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
13802024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
13812024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
13822024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
13832024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
13842024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
13852024-06-21T21:38:37.675ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
13862024-06-21T21:38:37.675ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 504762 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 123e72d0-5f8d-4756-804f-30e6c81ea0a1 response_code = 500 uri = /omicron-zones
13872024-06-21T21:38:37.675ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "123e72d0-5f8d-4756-804f-30e6c81ea0a1", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:37 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "123e72d0-5f8d-4756-804f-30e6c81ea0a1" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 249.008066ms sled_address = [fd00:1122:3344:101::1]:12345
13882024-06-21T21:38:37.925ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
13892024-06-21T21:38:37.925ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13902024-06-21T21:38:37.925ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
13912024-06-21T21:38:37.947ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13922024-06-21T21:38:37.947ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13932024-06-21T21:38:37.967ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
13942024-06-21T21:38:37.967ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
13952024-06-21T21:38:37.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13962024-06-21T21:38:37.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13972024-06-21T21:38:38.004ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
13982024-06-21T21:38:38.004ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
13992024-06-21T21:38:38.023ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14002024-06-21T21:38:38.023ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14012024-06-21T21:38:38.044ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14022024-06-21T21:38:38.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14032024-06-21T21:38:38.064ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14042024-06-21T21:38:38.064ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14052024-06-21T21:38:38.084ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14062024-06-21T21:38:38.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14072024-06-21T21:38:38.105ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14082024-06-21T21:38:38.105ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14092024-06-21T21:38:38.128ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14102024-06-21T21:38:38.128ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14112024-06-21T21:38:38.148ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14122024-06-21T21:38:38.148ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
14132024-06-21T21:38:38.169ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14142024-06-21T21:38:38.169ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
14152024-06-21T21:38:38.188ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14162024-06-21T21:38:38.188ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14172024-06-21T21:38:38.207ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14182024-06-21T21:38:38.207ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
14192024-06-21T21:38:38.226ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14202024-06-21T21:38:38.226ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
14212024-06-21T21:38:38.245ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14222024-06-21T21:38:38.245ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
14232024-06-21T21:38:38.264ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14242024-06-21T21:38:38.264ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
14252024-06-21T21:38:38.281ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14262024-06-21T21:38:38.281ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
14272024-06-21T21:38:38.300ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14282024-06-21T21:38:38.300ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
14292024-06-21T21:38:38.321ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14302024-06-21T21:38:38.321ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
14312024-06-21T21:38:38.321ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
14322024-06-21T21:38:38.321ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14332024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
14342024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
14352024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
14362024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
14372024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
14382024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
14392024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
14402024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
14412024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
14422024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
14432024-06-21T21:38:38.416ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
14442024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
14452024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
14462024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
14472024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
14482024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
14492024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
14502024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
14512024-06-21T21:38:38.417ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
14522024-06-21T21:38:38.417ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 491412 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 45e358d1-e139-47b0-a754-22fda10becc0 response_code = 500 uri = /omicron-zones
14532024-06-21T21:38:38.417ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "45e358d1-e139-47b0-a754-22fda10becc0", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:38 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "45e358d1-e139-47b0-a754-22fda10becc0" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 519.52932ms sled_address = [fd00:1122:3344:101::1]:12345
14542024-06-21T21:38:38.937ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
14552024-06-21T21:38:38.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14562024-06-21T21:38:38.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
14572024-06-21T21:38:38.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14582024-06-21T21:38:38.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14592024-06-21T21:38:38.984ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14602024-06-21T21:38:38.984ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
14612024-06-21T21:38:39.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14622024-06-21T21:38:39.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14632024-06-21T21:38:39.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14642024-06-21T21:38:39.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14652024-06-21T21:38:39.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14662024-06-21T21:38:39.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14672024-06-21T21:38:39.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14682024-06-21T21:38:39.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14692024-06-21T21:38:39.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
14702024-06-21T21:38:39.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
14712024-06-21T21:38:39.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14722024-06-21T21:38:39.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14732024-06-21T21:38:39.133ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
14742024-06-21T21:38:39.133ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
14752024-06-21T21:38:39.153ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
14762024-06-21T21:38:39.153ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
14772024-06-21T21:38:39.174ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14782024-06-21T21:38:39.174ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
14792024-06-21T21:38:39.196ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14802024-06-21T21:38:39.196ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
14812024-06-21T21:38:39.217ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14822024-06-21T21:38:39.217ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
14832024-06-21T21:38:39.239ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14842024-06-21T21:38:39.239ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
14852024-06-21T21:38:39.258ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14862024-06-21T21:38:39.258ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
14872024-06-21T21:38:39.279ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14882024-06-21T21:38:39.279ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
14892024-06-21T21:38:39.300ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14902024-06-21T21:38:39.300ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
14912024-06-21T21:38:39.319ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14922024-06-21T21:38:39.319ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
14932024-06-21T21:38:39.339ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
14942024-06-21T21:38:39.339ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
14952024-06-21T21:38:39.360ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14962024-06-21T21:38:39.360ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
14972024-06-21T21:38:39.360ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
14982024-06-21T21:38:39.361ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
14992024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
15002024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
15012024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
15022024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
15032024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
15042024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
15052024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
15062024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
15072024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
15082024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
15092024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
15102024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
15112024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
15122024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
15132024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
15142024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
15152024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
15162024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
15172024-06-21T21:38:39.461ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
15182024-06-21T21:38:39.461ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 523394 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = d87ff845-959d-44da-9ab3-eae3829e89ea response_code = 500 uri = /omicron-zones
15192024-06-21T21:38:39.461ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "d87ff845-959d-44da-9ab3-eae3829e89ea", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:39 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "d87ff845-959d-44da-9ab3-eae3829e89ea" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 307.094068ms sled_address = [fd00:1122:3344:101::1]:12345
15202024-06-21T21:38:39.589ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:61736
15212024-06-21T21:38:39.589ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 42 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:61736 req_id = f4cb3f97-33df-4d4e-9b42-7844fc5eb7ef response_code = 200 uri = /baseboard
15222024-06-21T21:38:39.770ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
15232024-06-21T21:38:39.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15242024-06-21T21:38:39.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
15252024-06-21T21:38:39.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15262024-06-21T21:38:39.793ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15272024-06-21T21:38:39.812ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15282024-06-21T21:38:39.812ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15292024-06-21T21:38:39.830ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15302024-06-21T21:38:39.830ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15312024-06-21T21:38:39.848ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15322024-06-21T21:38:39.848ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15332024-06-21T21:38:39.867ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15342024-06-21T21:38:39.867ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15352024-06-21T21:38:39.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15362024-06-21T21:38:39.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15372024-06-21T21:38:39.905ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15382024-06-21T21:38:39.905ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15392024-06-21T21:38:39.923ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15402024-06-21T21:38:39.923ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15412024-06-21T21:38:39.943ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15422024-06-21T21:38:39.943ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
15432024-06-21T21:38:39.963ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
15442024-06-21T21:38:39.963ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
15452024-06-21T21:38:39.986ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15462024-06-21T21:38:39.986ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
15472024-06-21T21:38:40.009ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15482024-06-21T21:38:40.009ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
15492024-06-21T21:38:40.032ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15502024-06-21T21:38:40.032ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
15512024-06-21T21:38:40.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15522024-06-21T21:38:40.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
15532024-06-21T21:38:40.075ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15542024-06-21T21:38:40.075ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
15552024-06-21T21:38:40.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15562024-06-21T21:38:40.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
15572024-06-21T21:38:40.116ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15582024-06-21T21:38:40.116ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
15592024-06-21T21:38:40.137ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15602024-06-21T21:38:40.137ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
15612024-06-21T21:38:40.157ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
15622024-06-21T21:38:40.157ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
15632024-06-21T21:38:40.177ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15642024-06-21T21:38:40.177ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
15652024-06-21T21:38:40.177ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
15662024-06-21T21:38:40.177ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
15672024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
15682024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
15692024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
15702024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
15712024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
15722024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
15732024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
15742024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
15752024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
15762024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
15772024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
15782024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
15792024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
15802024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
15812024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
15822024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
15832024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
15842024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
15852024-06-21T21:38:40.272ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
15862024-06-21T21:38:40.272ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 502745 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 4a7db128-3aa3-4fe3-baff-779680c796cc response_code = 500 uri = /omicron-zones
15872024-06-21T21:38:40.273ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4a7db128-3aa3-4fe3-baff-779680c796cc", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:40 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4a7db128-3aa3-4fe3-baff-779680c796cc" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 693.433005ms sled_address = [fd00:1122:3344:101::1]:12345
15882024-06-21T21:38:40.968ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
15892024-06-21T21:38:40.969ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15902024-06-21T21:38:40.969ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
15912024-06-21T21:38:40.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15922024-06-21T21:38:40.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15932024-06-21T21:38:41.013ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
15942024-06-21T21:38:41.013ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
15952024-06-21T21:38:41.038ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15962024-06-21T21:38:41.038ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15972024-06-21T21:38:41.058ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
15982024-06-21T21:38:41.058ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
15992024-06-21T21:38:41.080ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16002024-06-21T21:38:41.080ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16012024-06-21T21:38:41.100ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16022024-06-21T21:38:41.100ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16032024-06-21T21:38:41.121ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16042024-06-21T21:38:41.121ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16052024-06-21T21:38:41.127ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 1.077173041s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
16062024-06-21T21:38:41.143ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16072024-06-21T21:38:41.144ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16082024-06-21T21:38:41.163ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16092024-06-21T21:38:41.163ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16102024-06-21T21:38:41.184ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16112024-06-21T21:38:41.184ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16122024-06-21T21:38:41.203ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16132024-06-21T21:38:41.203ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
16142024-06-21T21:38:41.221ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16152024-06-21T21:38:41.221ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
16162024-06-21T21:38:41.241ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16172024-06-21T21:38:41.241ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16182024-06-21T21:38:41.259ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16192024-06-21T21:38:41.259ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
16202024-06-21T21:38:41.282ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16212024-06-21T21:38:41.282ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
16222024-06-21T21:38:41.304ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16232024-06-21T21:38:41.304ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
16242024-06-21T21:38:41.324ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16252024-06-21T21:38:41.324ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
16262024-06-21T21:38:41.343ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16272024-06-21T21:38:41.343ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
16282024-06-21T21:38:41.364ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16292024-06-21T21:38:41.364ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
16302024-06-21T21:38:41.383ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16312024-06-21T21:38:41.383ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
16322024-06-21T21:38:41.384ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
16332024-06-21T21:38:41.384ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16342024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
16352024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
16362024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
16372024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
16382024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
16392024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
16402024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
16412024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
16422024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
16432024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
16442024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
16452024-06-21T21:38:41.481ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
16462024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
16472024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
16482024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
16492024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
16502024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
16512024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
16522024-06-21T21:38:41.482ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
16532024-06-21T21:38:41.482ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 513530 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 94b28a91-fb92-4b03-9dbc-9f072c141199 response_code = 500 uri = /omicron-zones
16542024-06-21T21:38:41.482ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "94b28a91-fb92-4b03-9dbc-9f072c141199", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:40 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "94b28a91-fb92-4b03-9dbc-9f072c141199" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 521.220871ms sled_address = [fd00:1122:3344:101::1]:12345
16552024-06-21T21:38:42.004ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
16562024-06-21T21:38:42.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16572024-06-21T21:38:42.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
16582024-06-21T21:38:42.027ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16592024-06-21T21:38:42.027ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16602024-06-21T21:38:42.049ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16612024-06-21T21:38:42.049ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
16622024-06-21T21:38:42.068ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16632024-06-21T21:38:42.068ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16642024-06-21T21:38:42.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16652024-06-21T21:38:42.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16662024-06-21T21:38:42.110ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16672024-06-21T21:38:42.110ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16682024-06-21T21:38:42.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16692024-06-21T21:38:42.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16702024-06-21T21:38:42.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
16712024-06-21T21:38:42.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
16722024-06-21T21:38:42.171ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16732024-06-21T21:38:42.171ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16742024-06-21T21:38:42.193ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
16752024-06-21T21:38:42.193ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
16762024-06-21T21:38:42.213ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
16772024-06-21T21:38:42.213ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
16782024-06-21T21:38:42.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16792024-06-21T21:38:42.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
16802024-06-21T21:38:42.255ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16812024-06-21T21:38:42.255ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
16822024-06-21T21:38:42.275ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16832024-06-21T21:38:42.275ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
16842024-06-21T21:38:42.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16852024-06-21T21:38:42.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
16862024-06-21T21:38:42.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16872024-06-21T21:38:42.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
16882024-06-21T21:38:42.334ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16892024-06-21T21:38:42.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
16902024-06-21T21:38:42.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16912024-06-21T21:38:42.354ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
16922024-06-21T21:38:42.375ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16932024-06-21T21:38:42.375ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
16942024-06-21T21:38:42.395ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
16952024-06-21T21:38:42.395ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
16962024-06-21T21:38:42.413ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
16972024-06-21T21:38:42.413ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
16982024-06-21T21:38:42.413ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
16992024-06-21T21:38:42.413ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17002024-06-21T21:38:42.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
17012024-06-21T21:38:42.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
17022024-06-21T21:38:42.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
17032024-06-21T21:38:42.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
17042024-06-21T21:38:42.513ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
17052024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
17062024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
17072024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
17082024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
17092024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
17102024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
17112024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
17122024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
17132024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
17142024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
17152024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
17162024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
17172024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
17182024-06-21T21:38:42.514ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
17192024-06-21T21:38:42.514ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 509168 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = c80a9bfc-f975-4fe9-a1a3-1da107d4beb5 response_code = 500 uri = /omicron-zones
17202024-06-21T21:38:42.514ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c80a9bfc-f975-4fe9-a1a3-1da107d4beb5", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:42 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c80a9bfc-f975-4fe9-a1a3-1da107d4beb5" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 638.522127ms sled_address = [fd00:1122:3344:101::1]:12345
17212024-06-21T21:38:43.153ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
17222024-06-21T21:38:43.154ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17232024-06-21T21:38:43.154ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
17242024-06-21T21:38:43.176ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17252024-06-21T21:38:43.176ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17262024-06-21T21:38:43.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17272024-06-21T21:38:43.198ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17282024-06-21T21:38:43.220ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17292024-06-21T21:38:43.220ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17302024-06-21T21:38:43.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17312024-06-21T21:38:43.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17322024-06-21T21:38:43.265ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17332024-06-21T21:38:43.265ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17342024-06-21T21:38:43.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17352024-06-21T21:38:43.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17362024-06-21T21:38:43.307ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17372024-06-21T21:38:43.307ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17382024-06-21T21:38:43.327ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17392024-06-21T21:38:43.328ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17402024-06-21T21:38:43.348ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17412024-06-21T21:38:43.348ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
17422024-06-21T21:38:43.368ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
17432024-06-21T21:38:43.368ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
17442024-06-21T21:38:43.389ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17452024-06-21T21:38:43.389ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
17462024-06-21T21:38:43.409ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17472024-06-21T21:38:43.409ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
17482024-06-21T21:38:43.429ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17492024-06-21T21:38:43.429ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
17502024-06-21T21:38:43.449ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17512024-06-21T21:38:43.449ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
17522024-06-21T21:38:43.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17532024-06-21T21:38:43.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
17542024-06-21T21:38:43.491ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17552024-06-21T21:38:43.491ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
17562024-06-21T21:38:43.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17572024-06-21T21:38:43.512ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
17582024-06-21T21:38:43.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17592024-06-21T21:38:43.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
17602024-06-21T21:38:43.554ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
17612024-06-21T21:38:43.554ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
17622024-06-21T21:38:43.577ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17632024-06-21T21:38:43.577ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
17642024-06-21T21:38:43.578ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
17652024-06-21T21:38:43.578ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
17662024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
17672024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
17682024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
17692024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
17702024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
17712024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
17722024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
17732024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
17742024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
17752024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
17762024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
17772024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
17782024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
17792024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
17802024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
17812024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
17822024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
17832024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
17842024-06-21T21:38:43.681ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
17852024-06-21T21:38:43.681ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 527439 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = a5cfb007-f6ca-47da-9c9f-f73e42cacae7 response_code = 500 uri = /omicron-zones
17862024-06-21T21:38:43.681ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a5cfb007-f6ca-47da-9c9f-f73e42cacae7", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a5cfb007-f6ca-47da-9c9f-f73e42cacae7" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.475820776s sled_address = [fd00:1122:3344:101::1]:12345
17872024-06-21T21:38:45.158ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
17882024-06-21T21:38:45.158ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17892024-06-21T21:38:45.158ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
17902024-06-21T21:38:45.180ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17912024-06-21T21:38:45.180ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17922024-06-21T21:38:45.202ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
17932024-06-21T21:38:45.202ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
17942024-06-21T21:38:45.222ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17952024-06-21T21:38:45.222ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17962024-06-21T21:38:45.242ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17972024-06-21T21:38:45.242ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
17982024-06-21T21:38:45.263ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
17992024-06-21T21:38:45.263ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18002024-06-21T21:38:45.286ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18012024-06-21T21:38:45.286ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18022024-06-21T21:38:45.306ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18032024-06-21T21:38:45.306ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18042024-06-21T21:38:45.326ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18052024-06-21T21:38:45.326ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18062024-06-21T21:38:45.346ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18072024-06-21T21:38:45.346ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18082024-06-21T21:38:45.366ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18092024-06-21T21:38:45.366ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18102024-06-21T21:38:45.387ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18112024-06-21T21:38:45.387ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
18122024-06-21T21:38:45.408ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18132024-06-21T21:38:45.408ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
18142024-06-21T21:38:45.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18152024-06-21T21:38:45.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18162024-06-21T21:38:45.448ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18172024-06-21T21:38:45.448ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
18182024-06-21T21:38:45.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18192024-06-21T21:38:45.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
18202024-06-21T21:38:45.492ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18212024-06-21T21:38:45.492ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
18222024-06-21T21:38:45.512ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18232024-06-21T21:38:45.513ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
18242024-06-21T21:38:45.533ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18252024-06-21T21:38:45.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
18262024-06-21T21:38:45.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18272024-06-21T21:38:45.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
18282024-06-21T21:38:45.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18292024-06-21T21:38:45.574ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
18302024-06-21T21:38:45.574ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
18312024-06-21T21:38:45.574ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18322024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
18332024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
18342024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
18352024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
18362024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
18372024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
18382024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
18392024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
18402024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
18412024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
18422024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
18432024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
18442024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
18452024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
18462024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
18472024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
18482024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
18492024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
18502024-06-21T21:38:45.677ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
18512024-06-21T21:38:45.677ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 518969 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = a0b297ac-b192-460f-8907-879aa24cd4af response_code = 500 uri = /omicron-zones
18522024-06-21T21:38:45.677ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a0b297ac-b192-460f-8907-879aa24cd4af", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:45 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a0b297ac-b192-460f-8907-879aa24cd4af" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.576384694s sled_address = [fd00:1122:3344:101::1]:12345
18532024-06-21T21:38:47.255ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
18542024-06-21T21:38:47.256ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18552024-06-21T21:38:47.256ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
18562024-06-21T21:38:47.278ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18572024-06-21T21:38:47.278ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18582024-06-21T21:38:47.296ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18592024-06-21T21:38:47.296ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
18602024-06-21T21:38:47.315ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18612024-06-21T21:38:47.315ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18622024-06-21T21:38:47.334ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18632024-06-21T21:38:47.334ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18642024-06-21T21:38:47.355ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18652024-06-21T21:38:47.355ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18662024-06-21T21:38:47.376ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18672024-06-21T21:38:47.376ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18682024-06-21T21:38:47.396ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
18692024-06-21T21:38:47.396ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
18702024-06-21T21:38:47.418ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18712024-06-21T21:38:47.418ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18722024-06-21T21:38:47.442ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
18732024-06-21T21:38:47.442ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
18742024-06-21T21:38:47.464ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
18752024-06-21T21:38:47.464ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
18762024-06-21T21:38:47.485ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18772024-06-21T21:38:47.485ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
18782024-06-21T21:38:47.510ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18792024-06-21T21:38:47.510ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
18802024-06-21T21:38:47.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18812024-06-21T21:38:47.533ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
18822024-06-21T21:38:47.555ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18832024-06-21T21:38:47.555ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
18842024-06-21T21:38:47.577ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18852024-06-21T21:38:47.577ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
18862024-06-21T21:38:47.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18872024-06-21T21:38:47.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
18882024-06-21T21:38:47.621ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18892024-06-21T21:38:47.622ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
18902024-06-21T21:38:47.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18912024-06-21T21:38:47.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
18922024-06-21T21:38:47.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
18932024-06-21T21:38:47.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
18942024-06-21T21:38:47.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18952024-06-21T21:38:47.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
18962024-06-21T21:38:47.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
18972024-06-21T21:38:47.685ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
18982024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
18992024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
19002024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
19012024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
19022024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
19032024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
19042024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
19052024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
19062024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
19072024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
19082024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
19092024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
19102024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
19112024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
19122024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
19132024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
19142024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
19152024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
19162024-06-21T21:38:47.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
19172024-06-21T21:38:47.785ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 529840 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = a056a2af-57db-4087-acd1-8a69df8cbee1 response_code = 500 uri = /omicron-zones
19182024-06-21T21:38:47.785ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "a056a2af-57db-4087-acd1-8a69df8cbee1", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:47 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "a056a2af-57db-4087-acd1-8a69df8cbee1" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.500974807s sled_address = [fd00:1122:3344:101::1]:12345
19192024-06-21T21:38:49.287ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
19202024-06-21T21:38:49.287ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19212024-06-21T21:38:49.287ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
19222024-06-21T21:38:49.311ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19232024-06-21T21:38:49.311ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19242024-06-21T21:38:49.332ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19252024-06-21T21:38:49.332ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19262024-06-21T21:38:49.353ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19272024-06-21T21:38:49.353ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19282024-06-21T21:38:49.372ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19292024-06-21T21:38:49.372ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19302024-06-21T21:38:49.391ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19312024-06-21T21:38:49.391ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19322024-06-21T21:38:49.412ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19332024-06-21T21:38:49.412ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19342024-06-21T21:38:49.431ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19352024-06-21T21:38:49.431ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19362024-06-21T21:38:49.450ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19372024-06-21T21:38:49.450ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19382024-06-21T21:38:49.469ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19392024-06-21T21:38:49.469ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
19402024-06-21T21:38:49.490ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
19412024-06-21T21:38:49.490ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
19422024-06-21T21:38:49.511ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19432024-06-21T21:38:49.511ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
19442024-06-21T21:38:49.531ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19452024-06-21T21:38:49.531ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
19462024-06-21T21:38:49.553ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19472024-06-21T21:38:49.553ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
19482024-06-21T21:38:49.573ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19492024-06-21T21:38:49.573ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
19502024-06-21T21:38:49.591ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19512024-06-21T21:38:49.591ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
19522024-06-21T21:38:49.608ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19532024-06-21T21:38:49.608ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
19542024-06-21T21:38:49.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19552024-06-21T21:38:49.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
19562024-06-21T21:38:49.646ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19572024-06-21T21:38:49.646ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
19582024-06-21T21:38:49.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
19592024-06-21T21:38:49.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
19602024-06-21T21:38:49.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19612024-06-21T21:38:49.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
19622024-06-21T21:38:49.684ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
19632024-06-21T21:38:49.684ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
19642024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
19652024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
19662024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
19672024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
19682024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
19692024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
19702024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
19712024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
19722024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
19732024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
19742024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
19752024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
19762024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
19772024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
19782024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
19792024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
19802024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
19812024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
19822024-06-21T21:38:49.785ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
19832024-06-21T21:38:49.785ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 497757 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 8919c1a1-c78b-42e8-9587-00a4e8ba87f6 response_code = 500 uri = /omicron-zones
19842024-06-21T21:38:49.785ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "8919c1a1-c78b-42e8-9587-00a4e8ba87f6", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:49 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "8919c1a1-c78b-42e8-9587-00a4e8ba87f6" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 1.714234116s sled_address = [fd00:1122:3344:101::1]:12345
19852024-06-21T21:38:51.500ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
19862024-06-21T21:38:51.501ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19872024-06-21T21:38:51.501ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
19882024-06-21T21:38:51.522ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19892024-06-21T21:38:51.522ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19902024-06-21T21:38:51.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
19912024-06-21T21:38:51.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
19922024-06-21T21:38:51.563ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19932024-06-21T21:38:51.563ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19942024-06-21T21:38:51.581ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19952024-06-21T21:38:51.581ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19962024-06-21T21:38:51.604ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19972024-06-21T21:38:51.604ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
19982024-06-21T21:38:51.627ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
19992024-06-21T21:38:51.627ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20002024-06-21T21:38:51.648ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20012024-06-21T21:38:51.648ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20022024-06-21T21:38:51.669ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20032024-06-21T21:38:51.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20042024-06-21T21:38:51.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20052024-06-21T21:38:51.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20062024-06-21T21:38:51.712ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20072024-06-21T21:38:51.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20082024-06-21T21:38:51.732ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20092024-06-21T21:38:51.733ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
20102024-06-21T21:38:51.754ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20112024-06-21T21:38:51.754ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
20122024-06-21T21:38:51.774ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20132024-06-21T21:38:51.774ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20142024-06-21T21:38:51.793ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20152024-06-21T21:38:51.794ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
20162024-06-21T21:38:51.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20172024-06-21T21:38:51.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
20182024-06-21T21:38:51.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20192024-06-21T21:38:51.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
20202024-06-21T21:38:51.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20212024-06-21T21:38:51.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
20222024-06-21T21:38:51.876ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20232024-06-21T21:38:51.876ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
20242024-06-21T21:38:51.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20252024-06-21T21:38:51.897ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
20262024-06-21T21:38:51.916ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20272024-06-21T21:38:51.917ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
20282024-06-21T21:38:51.917ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
20292024-06-21T21:38:51.917ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20302024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
20312024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
20322024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
20332024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
20342024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
20352024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
20362024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
20372024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
20382024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
20392024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
20402024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
20412024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
20422024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
20432024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
20442024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
20452024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
20462024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
20472024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
20482024-06-21T21:38:52.017ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
20492024-06-21T21:38:52.017ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 516729 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 2724be82-99f6-4af0-a952-86f6da73c59b response_code = 500 uri = /omicron-zones
20502024-06-21T21:38:52.017ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "2724be82-99f6-4af0-a952-86f6da73c59b", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "2724be82-99f6-4af0-a952-86f6da73c59b" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 2.813672613s sled_address = [fd00:1122:3344:101::1]:12345
20512024-06-21T21:38:54.833ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
20522024-06-21T21:38:54.833ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20532024-06-21T21:38:54.833ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
20542024-06-21T21:38:54.856ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20552024-06-21T21:38:54.856ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20562024-06-21T21:38:54.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20572024-06-21T21:38:54.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
20582024-06-21T21:38:54.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20592024-06-21T21:38:54.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20602024-06-21T21:38:54.917ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20612024-06-21T21:38:54.917ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20622024-06-21T21:38:54.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20632024-06-21T21:38:54.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20642024-06-21T21:38:54.957ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20652024-06-21T21:38:54.957ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20662024-06-21T21:38:54.975ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
20672024-06-21T21:38:54.975ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
20682024-06-21T21:38:54.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20692024-06-21T21:38:54.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20702024-06-21T21:38:55.016ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
20712024-06-21T21:38:55.016ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
20722024-06-21T21:38:55.042ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
20732024-06-21T21:38:55.042ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
20742024-06-21T21:38:55.064ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20752024-06-21T21:38:55.064ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
20762024-06-21T21:38:55.085ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20772024-06-21T21:38:55.086ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
20782024-06-21T21:38:55.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20792024-06-21T21:38:55.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
20802024-06-21T21:38:55.126ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20812024-06-21T21:38:55.126ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
20822024-06-21T21:38:55.146ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20832024-06-21T21:38:55.146ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
20842024-06-21T21:38:55.166ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20852024-06-21T21:38:55.166ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
20862024-06-21T21:38:55.185ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20872024-06-21T21:38:55.185ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
20882024-06-21T21:38:55.205ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20892024-06-21T21:38:55.205ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
20902024-06-21T21:38:55.224ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
20912024-06-21T21:38:55.224ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
20922024-06-21T21:38:55.246ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20932024-06-21T21:38:55.246ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
20942024-06-21T21:38:55.246ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
20952024-06-21T21:38:55.247ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
20962024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
20972024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
20982024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
20992024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
21002024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
21012024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
21022024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
21032024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
21042024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
21052024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
21062024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
21072024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
21082024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
21092024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
21102024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
21112024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
21122024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
21132024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
21142024-06-21T21:38:55.341ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
21152024-06-21T21:38:55.341ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 508189 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = e458dedb-4ab5-4fb2-b14e-4541bb9b24e1 response_code = 500 uri = /omicron-zones
21162024-06-21T21:38:55.341ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e458dedb-4ab5-4fb2-b14e-4541bb9b24e1", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:54 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e458dedb-4ab5-4fb2-b14e-4541bb9b24e1" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 3.27726283s sled_address = [fd00:1122:3344:101::1]:12345
21172024-06-21T21:38:57.257ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 4.527545968s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
21182024-06-21T21:38:58.620ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
21192024-06-21T21:38:58.620ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21202024-06-21T21:38:58.620ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
21212024-06-21T21:38:58.644ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21222024-06-21T21:38:58.644ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21232024-06-21T21:38:58.664ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21242024-06-21T21:38:58.664ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21252024-06-21T21:38:58.685ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21262024-06-21T21:38:58.685ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21272024-06-21T21:38:58.704ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21282024-06-21T21:38:58.704ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21292024-06-21T21:38:58.723ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21302024-06-21T21:38:58.723ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21312024-06-21T21:38:58.743ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21322024-06-21T21:38:58.743ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21332024-06-21T21:38:58.762ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21342024-06-21T21:38:58.762ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21352024-06-21T21:38:58.781ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21362024-06-21T21:38:58.781ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21372024-06-21T21:38:58.804ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21382024-06-21T21:38:58.805ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
21392024-06-21T21:38:58.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
21402024-06-21T21:38:58.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
21412024-06-21T21:38:58.844ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21422024-06-21T21:38:58.844ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
21432024-06-21T21:38:58.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21442024-06-21T21:38:58.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
21452024-06-21T21:38:58.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21462024-06-21T21:38:58.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
21472024-06-21T21:38:58.899ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21482024-06-21T21:38:58.899ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
21492024-06-21T21:38:58.917ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21502024-06-21T21:38:58.917ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
21512024-06-21T21:38:58.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21522024-06-21T21:38:58.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
21532024-06-21T21:38:58.954ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21542024-06-21T21:38:58.954ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
21552024-06-21T21:38:58.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21562024-06-21T21:38:58.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
21572024-06-21T21:38:58.991ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
21582024-06-21T21:38:58.991ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
21592024-06-21T21:38:59.009ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21602024-06-21T21:38:59.009ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
21612024-06-21T21:38:59.010ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
21622024-06-21T21:38:59.010ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
21632024-06-21T21:38:59.103ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
21642024-06-21T21:38:59.103ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
21652024-06-21T21:38:59.103ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
21662024-06-21T21:38:59.103ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
21672024-06-21T21:38:59.103ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
21682024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
21692024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
21702024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
21712024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
21722024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
21732024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
21742024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
21752024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
21762024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
21772024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
21782024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
21792024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
21802024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
21812024-06-21T21:38:59.104ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
21822024-06-21T21:38:59.104ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 484077 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 6971d4f3-5d86-41ad-ac50-1511ca4ab87a response_code = 500 uri = /omicron-zones
21832024-06-21T21:38:59.104ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "6971d4f3-5d86-41ad-ac50-1511ca4ab87a", "content-length": "124", "date": "Fri, 21 Jun 2024 21:38:58 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "6971d4f3-5d86-41ad-ac50-1511ca4ab87a" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 2.303456572s sled_address = [fd00:1122:3344:101::1]:12345
21842024-06-21T21:39:01.296ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
21852024-06-21T21:39:01.297ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21862024-06-21T21:39:01.297ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
21872024-06-21T21:39:01.321ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21882024-06-21T21:39:01.321ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21892024-06-21T21:39:01.342ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
21902024-06-21T21:39:01.342ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
21912024-06-21T21:39:01.362ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21922024-06-21T21:39:01.362ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21932024-06-21T21:39:01.383ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21942024-06-21T21:39:01.383ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21952024-06-21T21:39:01.402ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21962024-06-21T21:39:01.402ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21972024-06-21T21:39:01.420ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
21982024-06-21T21:39:01.420ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
21992024-06-21T21:39:01.438ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22002024-06-21T21:39:01.438ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22012024-06-21T21:39:01.457ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22022024-06-21T21:39:01.457ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22032024-06-21T21:39:01.475ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22042024-06-21T21:39:01.475ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22052024-06-21T21:39:01.493ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
22062024-06-21T21:39:01.493ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22072024-06-21T21:39:01.513ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22082024-06-21T21:39:01.514ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
22092024-06-21T21:39:01.532ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22102024-06-21T21:39:01.532ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
22112024-06-21T21:39:01.551ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22122024-06-21T21:39:01.551ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
22132024-06-21T21:39:01.569ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22142024-06-21T21:39:01.569ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
22152024-06-21T21:39:01.587ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22162024-06-21T21:39:01.587ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
22172024-06-21T21:39:01.605ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22182024-06-21T21:39:01.605ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
22192024-06-21T21:39:01.623ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22202024-06-21T21:39:01.623ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
22212024-06-21T21:39:01.643ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22222024-06-21T21:39:01.643ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
22232024-06-21T21:39:01.661ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22242024-06-21T21:39:01.661ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
22252024-06-21T21:39:01.680ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22262024-06-21T21:39:01.680ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
22272024-06-21T21:39:01.680ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
22282024-06-21T21:39:01.680ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22292024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
22302024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
22312024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
22322024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
22332024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
22342024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
22352024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
22362024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
22372024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
22382024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
22392024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
22402024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
22412024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
22422024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
22432024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
22442024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
22452024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
22462024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
22472024-06-21T21:39:01.779ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
22482024-06-21T21:39:01.779ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 482482 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 4f0b2f10-dffb-450e-bd0f-d285de807755 response_code = 500 uri = /omicron-zones
22492024-06-21T21:39:01.779ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "4f0b2f10-dffb-450e-bd0f-d285de807755", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:01 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "4f0b2f10-dffb-450e-bd0f-d285de807755" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 3.001349973s sled_address = [fd00:1122:3344:101::1]:12345
22502024-06-21T21:39:04.774ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
22512024-06-21T21:39:04.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22522024-06-21T21:39:04.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
22532024-06-21T21:39:04.797ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22542024-06-21T21:39:04.799ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22552024-06-21T21:39:04.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22562024-06-21T21:39:04.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
22572024-06-21T21:39:04.837ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22582024-06-21T21:39:04.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22592024-06-21T21:39:04.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22602024-06-21T21:39:04.857ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22612024-06-21T21:39:04.875ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22622024-06-21T21:39:04.875ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22632024-06-21T21:39:04.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22642024-06-21T21:39:04.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22652024-06-21T21:39:04.912ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
22662024-06-21T21:39:04.912ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
22672024-06-21T21:39:04.931ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22682024-06-21T21:39:04.931ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22692024-06-21T21:39:04.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
22702024-06-21T21:39:04.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
22712024-06-21T21:39:04.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
22722024-06-21T21:39:04.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
22732024-06-21T21:39:04.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22742024-06-21T21:39:04.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
22752024-06-21T21:39:05.005ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22762024-06-21T21:39:05.005ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
22772024-06-21T21:39:05.028ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22782024-06-21T21:39:05.028ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
22792024-06-21T21:39:05.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22802024-06-21T21:39:05.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
22812024-06-21T21:39:05.066ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22822024-06-21T21:39:05.066ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
22832024-06-21T21:39:05.083ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22842024-06-21T21:39:05.084ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
22852024-06-21T21:39:05.102ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22862024-06-21T21:39:05.102ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
22872024-06-21T21:39:05.120ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22882024-06-21T21:39:05.120ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
22892024-06-21T21:39:05.138ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
22902024-06-21T21:39:05.138ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
22912024-06-21T21:39:05.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22922024-06-21T21:39:05.157ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
22932024-06-21T21:39:05.157ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
22942024-06-21T21:39:05.157ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
22952024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
22962024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
22972024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
22982024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
22992024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
23002024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
23012024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
23022024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
23032024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
23042024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
23052024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
23062024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
23072024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
23082024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
23092024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
23102024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
23112024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
23122024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
23132024-06-21T21:39:05.251ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
23142024-06-21T21:39:05.251ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 478401 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 9591452e-94d7-4a2e-b4b7-90234d9ccf57 response_code = 500 uri = /omicron-zones
23152024-06-21T21:39:05.252ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "9591452e-94d7-4a2e-b4b7-90234d9ccf57", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:05 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "9591452e-94d7-4a2e-b4b7-90234d9ccf57" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 2.472152315s sled_address = [fd00:1122:3344:101::1]:12345
23162024-06-21T21:39:07.718ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
23172024-06-21T21:39:07.718ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23182024-06-21T21:39:07.719ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
23192024-06-21T21:39:07.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23202024-06-21T21:39:07.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23212024-06-21T21:39:07.759ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23222024-06-21T21:39:07.760ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23232024-06-21T21:39:07.780ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23242024-06-21T21:39:07.780ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23252024-06-21T21:39:07.798ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23262024-06-21T21:39:07.798ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23272024-06-21T21:39:07.817ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23282024-06-21T21:39:07.817ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23292024-06-21T21:39:07.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23302024-06-21T21:39:07.836ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23312024-06-21T21:39:07.854ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23322024-06-21T21:39:07.854ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23332024-06-21T21:39:07.872ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23342024-06-21T21:39:07.872ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23352024-06-21T21:39:07.891ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23362024-06-21T21:39:07.891ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
23372024-06-21T21:39:07.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
23382024-06-21T21:39:07.910ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
23392024-06-21T21:39:07.928ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23402024-06-21T21:39:07.928ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
23412024-06-21T21:39:07.949ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23422024-06-21T21:39:07.949ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
23432024-06-21T21:39:07.968ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23442024-06-21T21:39:07.968ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
23452024-06-21T21:39:07.986ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23462024-06-21T21:39:07.986ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
23472024-06-21T21:39:08.004ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23482024-06-21T21:39:08.004ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
23492024-06-21T21:39:08.025ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23502024-06-21T21:39:08.025ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
23512024-06-21T21:39:08.043ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23522024-06-21T21:39:08.044ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
23532024-06-21T21:39:08.062ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23542024-06-21T21:39:08.062ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
23552024-06-21T21:39:08.079ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
23562024-06-21T21:39:08.079ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
23572024-06-21T21:39:08.098ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23582024-06-21T21:39:08.098ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
23592024-06-21T21:39:08.099ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
23602024-06-21T21:39:08.099ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
23612024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
23622024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
23632024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
23642024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
23652024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
23662024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
23672024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
23682024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
23692024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
23702024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
23712024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
23722024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
23732024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
23742024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
23752024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
23762024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
23772024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
23782024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
23792024-06-21T21:39:08.191ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
23802024-06-21T21:39:08.191ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 474145 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 227b936b-c2cb-4d7b-a6c6-85af565b2e38 response_code = 500 uri = /omicron-zones
23812024-06-21T21:39:08.191ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "227b936b-c2cb-4d7b-a6c6-85af565b2e38", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:08 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "227b936b-c2cb-4d7b-a6c6-85af565b2e38" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 6.608933287s sled_address = [fd00:1122:3344:101::1]:12345
23822024-06-21T21:39:09.490ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:48610
23832024-06-21T21:39:09.490ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 50 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:48610 req_id = fd926ab3-ad59-4ca1-8bec-bc8805f8833e response_code = 200 uri = /baseboard
23842024-06-21T21:39:14.783ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
23852024-06-21T21:39:14.784ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23862024-06-21T21:39:14.784ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
23872024-06-21T21:39:14.806ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23882024-06-21T21:39:14.806ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23892024-06-21T21:39:14.825ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
23902024-06-21T21:39:14.825ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
23912024-06-21T21:39:14.846ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23922024-06-21T21:39:14.846ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23932024-06-21T21:39:14.864ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23942024-06-21T21:39:14.864ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23952024-06-21T21:39:14.883ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23962024-06-21T21:39:14.883ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23972024-06-21T21:39:14.900ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
23982024-06-21T21:39:14.900ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
23992024-06-21T21:39:14.920ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24002024-06-21T21:39:14.920ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24012024-06-21T21:39:14.938ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24022024-06-21T21:39:14.938ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24032024-06-21T21:39:14.959ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24042024-06-21T21:39:14.961ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24052024-06-21T21:39:14.979ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
24062024-06-21T21:39:14.979ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24072024-06-21T21:39:14.998ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24082024-06-21T21:39:14.998ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
24092024-06-21T21:39:15.018ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24102024-06-21T21:39:15.018ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
24112024-06-21T21:39:15.037ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24122024-06-21T21:39:15.037ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
24132024-06-21T21:39:15.057ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24142024-06-21T21:39:15.057ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
24152024-06-21T21:39:15.076ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24162024-06-21T21:39:15.076ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
24172024-06-21T21:39:15.095ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24182024-06-21T21:39:15.095ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
24192024-06-21T21:39:15.112ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24202024-06-21T21:39:15.112ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
24212024-06-21T21:39:15.129ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24222024-06-21T21:39:15.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
24232024-06-21T21:39:15.148ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24242024-06-21T21:39:15.148ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
24252024-06-21T21:39:15.166ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24262024-06-21T21:39:15.166ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
24272024-06-21T21:39:15.166ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
24282024-06-21T21:39:15.167ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24292024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
24302024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
24312024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
24322024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
24332024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
24342024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
24352024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
24362024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
24372024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
24382024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
24392024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
24402024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
24412024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
24422024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
24432024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
24442024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
24452024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
24462024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
24472024-06-21T21:39:15.259ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
24482024-06-21T21:39:15.259ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 477281 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 54185345-2720-4b38-8ed5-ffee3a3e1383 response_code = 500 uri = /omicron-zones
24492024-06-21T21:39:15.260ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "54185345-2720-4b38-8ed5-ffee3a3e1383", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:15 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "54185345-2720-4b38-8ed5-ffee3a3e1383" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 9.168325066s sled_address = [fd00:1122:3344:101::1]:12345
24502024-06-21T21:39:16.725ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 8.944880162s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
24512024-06-21T21:39:24.404ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
24522024-06-21T21:39:24.405ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24532024-06-21T21:39:24.405ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
24542024-06-21T21:39:24.428ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24552024-06-21T21:39:24.428ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24562024-06-21T21:39:24.450ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24572024-06-21T21:39:24.450ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
24582024-06-21T21:39:24.470ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24592024-06-21T21:39:24.470ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24602024-06-21T21:39:24.489ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24612024-06-21T21:39:24.489ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24622024-06-21T21:39:24.508ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24632024-06-21T21:39:24.508ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24642024-06-21T21:39:24.526ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24652024-06-21T21:39:24.526ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24662024-06-21T21:39:24.544ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
24672024-06-21T21:39:24.544ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
24682024-06-21T21:39:24.562ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24692024-06-21T21:39:24.562ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24702024-06-21T21:39:24.580ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
24712024-06-21T21:39:24.580ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
24722024-06-21T21:39:24.599ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
24732024-06-21T21:39:24.599ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
24742024-06-21T21:39:24.618ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24752024-06-21T21:39:24.618ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
24762024-06-21T21:39:24.637ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24772024-06-21T21:39:24.637ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
24782024-06-21T21:39:24.656ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24792024-06-21T21:39:24.656ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
24802024-06-21T21:39:24.674ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24812024-06-21T21:39:24.674ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
24822024-06-21T21:39:24.692ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24832024-06-21T21:39:24.692ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
24842024-06-21T21:39:24.710ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24852024-06-21T21:39:24.712ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
24862024-06-21T21:39:24.729ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24872024-06-21T21:39:24.729ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
24882024-06-21T21:39:24.747ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24892024-06-21T21:39:24.747ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
24902024-06-21T21:39:24.765ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
24912024-06-21T21:39:24.765ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
24922024-06-21T21:39:24.783ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24932024-06-21T21:39:24.783ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
24942024-06-21T21:39:24.783ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
24952024-06-21T21:39:24.784ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
24962024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
24972024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
24982024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
24992024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
25002024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
25012024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
25022024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
25032024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
25042024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
25052024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
25062024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
25072024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
25082024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
25092024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
25102024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
25112024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
25122024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
25132024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
25142024-06-21T21:39:24.878ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
25152024-06-21T21:39:24.878ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 474594 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 7ed224d9-31b6-49a3-a5aa-741b46877794 response_code = 500 uri = /omicron-zones
25162024-06-21T21:39:24.878ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "7ed224d9-31b6-49a3-a5aa-741b46877794", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:24 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "7ed224d9-31b6-49a3-a5aa-741b46877794" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 8.806764964s sled_address = [fd00:1122:3344:101::1]:12345
25172024-06-21T21:39:33.661ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
25182024-06-21T21:39:33.662ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25192024-06-21T21:39:33.662ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
25202024-06-21T21:39:33.685ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25212024-06-21T21:39:33.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25222024-06-21T21:39:33.707ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25232024-06-21T21:39:33.709ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25242024-06-21T21:39:33.730ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25252024-06-21T21:39:33.730ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25262024-06-21T21:39:33.752ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25272024-06-21T21:39:33.752ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25282024-06-21T21:39:33.775ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25292024-06-21T21:39:33.775ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25302024-06-21T21:39:33.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25312024-06-21T21:39:33.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25322024-06-21T21:39:33.816ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25332024-06-21T21:39:33.816ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25342024-06-21T21:39:33.836ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25352024-06-21T21:39:33.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25362024-06-21T21:39:33.857ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25372024-06-21T21:39:33.858ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
25382024-06-21T21:39:33.878ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
25392024-06-21T21:39:33.878ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
25402024-06-21T21:39:33.897ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25412024-06-21T21:39:33.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
25422024-06-21T21:39:33.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25432024-06-21T21:39:33.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
25442024-06-21T21:39:33.937ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25452024-06-21T21:39:33.937ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
25462024-06-21T21:39:33.960ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25472024-06-21T21:39:33.960ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
25482024-06-21T21:39:33.985ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25492024-06-21T21:39:33.985ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
25502024-06-21T21:39:34.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25512024-06-21T21:39:34.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
25522024-06-21T21:39:34.031ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25532024-06-21T21:39:34.031ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
25542024-06-21T21:39:34.054ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25552024-06-21T21:39:34.054ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
25562024-06-21T21:39:34.077ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
25572024-06-21T21:39:34.077ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
25582024-06-21T21:39:34.099ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25592024-06-21T21:39:34.099ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
25602024-06-21T21:39:34.099ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
25612024-06-21T21:39:34.099ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
25622024-06-21T21:39:34.199ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
25632024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
25642024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
25652024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
25662024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
25672024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
25682024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
25692024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
25702024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
25712024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
25722024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
25732024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
25742024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
25752024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
25762024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
25772024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
25782024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
25792024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
25802024-06-21T21:39:34.200ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
25812024-06-21T21:39:34.200ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 539745 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = e35f8e3b-45a7-4cc0-8edd-76d7ad55dcb4 response_code = 500 uri = /omicron-zones
25822024-06-21T21:39:34.200ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "e35f8e3b-45a7-4cc0-8edd-76d7ad55dcb4", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:33 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "e35f8e3b-45a7-4cc0-8edd-76d7ad55dcb4" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 9.47666852s sled_address = [fd00:1122:3344:101::1]:12345
25832024-06-21T21:39:39.442ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:41208
25842024-06-21T21:39:39.442ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 72 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:41208 req_id = d005ca1c-6e26-4dd2-bf41-bf482573d907 response_code = 200 uri = /baseboard
25852024-06-21T21:39:40.641ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 8.856853868s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
25862024-06-21T21:39:43.652ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
25872024-06-21T21:39:43.654ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25882024-06-21T21:39:43.654ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
25892024-06-21T21:39:43.676ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25902024-06-21T21:39:43.676ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25912024-06-21T21:39:43.696ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
25922024-06-21T21:39:43.696ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
25932024-06-21T21:39:43.716ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25942024-06-21T21:39:43.716ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25952024-06-21T21:39:43.734ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25962024-06-21T21:39:43.734ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25972024-06-21T21:39:43.751ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
25982024-06-21T21:39:43.751ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
25992024-06-21T21:39:43.769ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26002024-06-21T21:39:43.769ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26012024-06-21T21:39:43.788ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26022024-06-21T21:39:43.788ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26032024-06-21T21:39:43.805ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26042024-06-21T21:39:43.805ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26052024-06-21T21:39:43.824ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26062024-06-21T21:39:43.824ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26072024-06-21T21:39:43.843ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
26082024-06-21T21:39:43.843ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26092024-06-21T21:39:43.861ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26102024-06-21T21:39:43.861ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
26112024-06-21T21:39:43.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26122024-06-21T21:39:43.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
26132024-06-21T21:39:43.898ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26142024-06-21T21:39:43.898ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
26152024-06-21T21:39:43.916ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26162024-06-21T21:39:43.916ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
26172024-06-21T21:39:43.935ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26182024-06-21T21:39:43.935ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
26192024-06-21T21:39:43.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26202024-06-21T21:39:43.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
26212024-06-21T21:39:43.972ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26222024-06-21T21:39:43.972ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
26232024-06-21T21:39:43.990ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26242024-06-21T21:39:43.990ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
26252024-06-21T21:39:44.008ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26262024-06-21T21:39:44.008ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
26272024-06-21T21:39:44.027ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26282024-06-21T21:39:44.027ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
26292024-06-21T21:39:44.028ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
26302024-06-21T21:39:44.028ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26312024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
26322024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
26332024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
26342024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
26352024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
26362024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
26372024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
26382024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
26392024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
26402024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
26412024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
26422024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
26432024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
26442024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
26452024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
26462024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
26472024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
26482024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
26492024-06-21T21:39:44.121ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
26502024-06-21T21:39:44.121ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 469577 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 322036d2-017c-42f0-90b5-cc69fb82b14e response_code = 500 uri = /omicron-zones
26512024-06-21T21:39:44.121ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "322036d2-017c-42f0-90b5-cc69fb82b14e", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:43 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "322036d2-017c-42f0-90b5-cc69fb82b14e" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 7.471388365s sled_address = [fd00:1122:3344:101::1]:12345
26522024-06-21T21:39:51.573ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
26532024-06-21T21:39:51.573ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26542024-06-21T21:39:51.573ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
26552024-06-21T21:39:51.597ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26562024-06-21T21:39:51.598ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26572024-06-21T21:39:51.617ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26582024-06-21T21:39:51.617ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
26592024-06-21T21:39:51.636ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26602024-06-21T21:39:51.636ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26612024-06-21T21:39:51.655ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26622024-06-21T21:39:51.655ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26632024-06-21T21:39:51.672ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26642024-06-21T21:39:51.672ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26652024-06-21T21:39:51.690ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26662024-06-21T21:39:51.690ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26672024-06-21T21:39:51.708ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
26682024-06-21T21:39:51.708ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
26692024-06-21T21:39:51.726ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26702024-06-21T21:39:51.726ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26712024-06-21T21:39:51.744ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
26722024-06-21T21:39:51.744ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
26732024-06-21T21:39:51.763ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
26742024-06-21T21:39:51.763ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
26752024-06-21T21:39:51.782ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26762024-06-21T21:39:51.782ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
26772024-06-21T21:39:51.800ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26782024-06-21T21:39:51.800ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
26792024-06-21T21:39:51.818ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26802024-06-21T21:39:51.818ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
26812024-06-21T21:39:51.837ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26822024-06-21T21:39:51.837ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
26832024-06-21T21:39:51.855ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26842024-06-21T21:39:51.855ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
26852024-06-21T21:39:51.874ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26862024-06-21T21:39:51.874ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
26872024-06-21T21:39:51.894ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26882024-06-21T21:39:51.894ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
26892024-06-21T21:39:51.911ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26902024-06-21T21:39:51.911ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
26912024-06-21T21:39:51.930ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
26922024-06-21T21:39:51.930ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
26932024-06-21T21:39:51.948ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26942024-06-21T21:39:51.948ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
26952024-06-21T21:39:51.948ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
26962024-06-21T21:39:51.948ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
26972024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
26982024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
26992024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
27002024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
27012024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
27022024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
27032024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
27042024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
27052024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
27062024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
27072024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
27082024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
27092024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
27102024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
27112024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
27122024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
27132024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
27142024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
27152024-06-21T21:39:52.039ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
27162024-06-21T21:39:52.039ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 467442 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = da0d5207-e4ae-4664-a82e-7659f94a05e4 response_code = 500 uri = /omicron-zones
27172024-06-21T21:39:52.040ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "da0d5207-e4ae-4664-a82e-7659f94a05e4", "content-length": "124", "date": "Fri, 21 Jun 2024 21:39:51 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "da0d5207-e4ae-4664-a82e-7659f94a05e4" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 11.596348581s sled_address = [fd00:1122:3344:101::1]:12345
27182024-06-21T21:40:03.605ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
27192024-06-21T21:40:03.606ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27202024-06-21T21:40:03.606ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
27212024-06-21T21:40:03.628ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27222024-06-21T21:40:03.628ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27232024-06-21T21:40:03.649ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27242024-06-21T21:40:03.649ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27252024-06-21T21:40:03.668ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27262024-06-21T21:40:03.669ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27272024-06-21T21:40:03.687ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27282024-06-21T21:40:03.687ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27292024-06-21T21:40:03.705ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27302024-06-21T21:40:03.705ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27312024-06-21T21:40:03.723ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27322024-06-21T21:40:03.723ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27332024-06-21T21:40:03.741ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27342024-06-21T21:40:03.741ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27352024-06-21T21:40:03.758ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27362024-06-21T21:40:03.758ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27372024-06-21T21:40:03.777ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27382024-06-21T21:40:03.777ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
27392024-06-21T21:40:03.796ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
27402024-06-21T21:40:03.796ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
27412024-06-21T21:40:03.814ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27422024-06-21T21:40:03.814ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
27432024-06-21T21:40:03.832ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27442024-06-21T21:40:03.832ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
27452024-06-21T21:40:03.852ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27462024-06-21T21:40:03.852ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
27472024-06-21T21:40:03.871ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27482024-06-21T21:40:03.871ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
27492024-06-21T21:40:03.890ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27502024-06-21T21:40:03.890ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
27512024-06-21T21:40:03.909ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27522024-06-21T21:40:03.909ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
27532024-06-21T21:40:03.927ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27542024-06-21T21:40:03.927ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
27552024-06-21T21:40:03.946ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27562024-06-21T21:40:03.946ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
27572024-06-21T21:40:03.964ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
27582024-06-21T21:40:03.964ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
27592024-06-21T21:40:03.982ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
27602024-06-21T21:40:03.982ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
27612024-06-21T21:40:03.983ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
27622024-06-21T21:40:03.983ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
27632024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
27642024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
27652024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
27662024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
27672024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
27682024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
27692024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
27702024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
27712024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
27722024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
27732024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
27742024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
27752024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
27762024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
27772024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
27782024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
27792024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
27802024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
27812024-06-21T21:40:04.075ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
27822024-06-21T21:40:04.075ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 470809 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = b19a1c97-6acf-42c7-bd74-572df69cd2aa response_code = 500 uri = /omicron-zones
27832024-06-21T21:40:04.075ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b19a1c97-6acf-42c7-bd74-572df69cd2aa", "content-length": "124", "date": "Fri, 21 Jun 2024 21:40:03 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "b19a1c97-6acf-42c7-bd74-572df69cd2aa" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 15.845874026s sled_address = [fd00:1122:3344:101::1]:12345
27842024-06-21T21:40:04.467ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 35.786617309s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
27852024-06-21T21:40:09.392ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:57825
27862024-06-21T21:40:09.393ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 810 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:57825 req_id = a3b0d1e8-136d-4f7a-bd0e-ddfa637c8dc0 response_code = 200 uri = /baseboard
27872024-06-21T21:40:19.880ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
27882024-06-21T21:40:19.880ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27892024-06-21T21:40:19.880ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
27902024-06-21T21:40:19.901ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27912024-06-21T21:40:19.901ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27922024-06-21T21:40:19.921ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
27932024-06-21T21:40:19.921ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
27942024-06-21T21:40:19.940ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27952024-06-21T21:40:19.940ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27962024-06-21T21:40:19.959ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27972024-06-21T21:40:19.959ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
27982024-06-21T21:40:19.977ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
27992024-06-21T21:40:19.977ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28002024-06-21T21:40:19.995ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28012024-06-21T21:40:19.995ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28022024-06-21T21:40:20.014ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28032024-06-21T21:40:20.014ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28042024-06-21T21:40:20.032ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28052024-06-21T21:40:20.032ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28062024-06-21T21:40:20.050ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28072024-06-21T21:40:20.050ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28082024-06-21T21:40:20.070ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
28092024-06-21T21:40:20.070ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
28102024-06-21T21:40:20.089ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28112024-06-21T21:40:20.089ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
28122024-06-21T21:40:20.106ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28132024-06-21T21:40:20.106ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
28142024-06-21T21:40:20.125ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28152024-06-21T21:40:20.125ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
28162024-06-21T21:40:20.143ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28172024-06-21T21:40:20.143ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
28182024-06-21T21:40:20.160ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28192024-06-21T21:40:20.160ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
28202024-06-21T21:40:20.179ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28212024-06-21T21:40:20.179ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
28222024-06-21T21:40:20.198ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28232024-06-21T21:40:20.198ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
28242024-06-21T21:40:20.216ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28252024-06-21T21:40:20.216ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
28262024-06-21T21:40:20.234ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28272024-06-21T21:40:20.234ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
28282024-06-21T21:40:20.252ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28292024-06-21T21:40:20.252ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
28302024-06-21T21:40:20.252ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
28312024-06-21T21:40:20.252ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28322024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
28332024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
28342024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
28352024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
28362024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
28372024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
28382024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
28392024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
28402024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
28412024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
28422024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
28432024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
28442024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
28452024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
28462024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
28472024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
28482024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
28492024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
28502024-06-21T21:40:20.343ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
28512024-06-21T21:40:20.343ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 464607 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = c914dc52-3b57-493f-9792-bcaf5e911dbc response_code = 500 uri = /omicron-zones
28522024-06-21T21:40:20.344ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "c914dc52-3b57-493f-9792-bcaf5e911dbc", "content-length": "124", "date": "Fri, 21 Jun 2024 21:40:20 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "c914dc52-3b57-493f-9792-bcaf5e911dbc" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 15.333558524s sled_address = [fd00:1122:3344:101::1]:12345
28532024-06-21T21:40:35.637ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
28542024-06-21T21:40:35.638ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28552024-06-21T21:40:35.638ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
28562024-06-21T21:40:35.663ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28572024-06-21T21:40:35.663ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
28582024-06-21T21:40:35.686ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28592024-06-21T21:40:35.686ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
28602024-06-21T21:40:35.706ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28612024-06-21T21:40:35.706ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28622024-06-21T21:40:35.727ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28632024-06-21T21:40:35.727ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28642024-06-21T21:40:35.747ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28652024-06-21T21:40:35.747ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28662024-06-21T21:40:35.767ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28672024-06-21T21:40:35.767ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28682024-06-21T21:40:35.786ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
28692024-06-21T21:40:35.786ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
28702024-06-21T21:40:35.807ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28712024-06-21T21:40:35.807ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28722024-06-21T21:40:35.827ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
28732024-06-21T21:40:35.827ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
28742024-06-21T21:40:35.847ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
28752024-06-21T21:40:35.847ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
28762024-06-21T21:40:35.868ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28772024-06-21T21:40:35.868ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
28782024-06-21T21:40:35.886ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28792024-06-21T21:40:35.886ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
28802024-06-21T21:40:35.905ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28812024-06-21T21:40:35.905ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
28822024-06-21T21:40:35.926ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28832024-06-21T21:40:35.926ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
28842024-06-21T21:40:35.947ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28852024-06-21T21:40:35.947ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
28862024-06-21T21:40:35.967ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28872024-06-21T21:40:35.967ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
28882024-06-21T21:40:35.987ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28892024-06-21T21:40:35.987ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
28902024-06-21T21:40:36.007ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28912024-06-21T21:40:36.007ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
28922024-06-21T21:40:36.029ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
28932024-06-21T21:40:36.029ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
28942024-06-21T21:40:36.049ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28952024-06-21T21:40:36.049ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
28962024-06-21T21:40:36.049ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
28972024-06-21T21:40:36.049ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
28982024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
28992024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
29002024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
29012024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
29022024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
29032024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
29042024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
29052024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
29062024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
29072024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
29082024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
29092024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
29102024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
29112024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
29122024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
29132024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
29142024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
29152024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
29162024-06-21T21:40:36.144ZWARNSledAgent (ServiceManager): Zone failed to start file = sled-agent/src/services.rs:2960 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
29172024-06-21T21:40:36.144ZINFOSledAgent (dropshot (SledAgent)): request completed error_message_external = Internal Server Error error_message_internal = Failed to initialize zones: [("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5", TimeNotSynchronized), ("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6", TimeNotSynchronized), ("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b", TimeNotSynchronized), ("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d", TimeNotSynchronized), ("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868", TimeNotSynchronized), ("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679", TimeNotSynchronized), ("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280", TimeNotSynchronized), ("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07", TimeNotSynchronized), ("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15", TimeNotSynchronized), ("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2", TimeNotSynchronized), ("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57", TimeNotSynchronized), ("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8", TimeNotSynchronized), ("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a", TimeNotSynchronized), ("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae", TimeNotSynchronized), ("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6", TimeNotSynchronized), ("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6", TimeNotSynchronized), ("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf", TimeNotSynchronized), ("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91", TimeNotSynchronized), ("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76", TimeNotSynchronized)] file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:866 latency_us = 508953 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = f5f8a949-046e-45cc-8d6f-98954f318c82 response_code = 500 uri = /omicron-zones
29182024-06-21T21:40:36.145ZWARNSledAgent (RSS): failed to initialize Omicron zones error = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "f5f8a949-046e-45cc-8d6f-98954f318c82", "content-length": "124", "date": "Fri, 21 Jun 2024 21:40:35 GMT"}; value: Error { error_code: Some("Internal"), message: "Internal Server Error", request_id: "f5f8a949-046e-45cc-8d6f-98954f318c82" } file = sled-agent/src/rack_setup/service.rs:469 retry_after = 21.660533436s sled_address = [fd00:1122:3344:101::1]:12345
29192024-06-21T21:40:39.343ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:43859
29202024-06-21T21:40:39.343ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 539 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:43859 req_id = 8258dfdc-3dfd-4fd3-8e99-3a95a0ae3276 response_code = 200 uri = /baseboard
29212024-06-21T21:40:55.149ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 37.150103644s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): operation timed out" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
29222024-06-21T21:40:57.748ZINFOSledAgent (RSS): attempting to set up sled's Omicron zones: OmicronZonesConfig { generation: Generation(5), zones: [OmicronZoneConfig { id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, underlay_address: fd00:1122:3344:1::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:1::1]:5353, dns_address: [fd00:1122:3344:1::1]:53, gz_address: fd00:1122:3344:1::2, gz_address_index: 0 } }, OmicronZoneConfig { id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, underlay_address: fd00:1122:3344:2::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:2::1]:5353, dns_address: [fd00:1122:3344:2::1]:53, gz_address: fd00:1122:3344:2::2, gz_address_index: 1 } }, OmicronZoneConfig { id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, underlay_address: fd00:1122:3344:3::1, zone_type: InternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } }, http_address: [fd00:1122:3344:3::1]:5353, dns_address: [fd00:1122:3344:3::1]:53, gz_address: fd00:1122:3344:3::2, gz_address_index: 2 } }, OmicronZoneConfig { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8, underlay_address: fd00:1122:3344:101::1b, zone_type: BoundaryNtp { address: [fd00:1122:3344:101::1b]:123, ntp_servers: ["0.pool.ntp.org"], dns_servers: [1.1.1.1, 9.9.9.9], domain: None, nic: NetworkInterface { id: 8d5eb02b-9914-404f-88f5-5b0790974f59, kind: Service { id: 0516b982-c4cb-4196-b8d1-f13d36b135a8 }, name: Name("ntp-0516b982-c4cb-4196-b8d1-f13d36b135a8"), ip: 172.30.3.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 205, 184])), subnet: V4(Ipv4Net { addr: 172.30.3.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, snat_cfg: SourceNatConfig { ip: 10.151.2.169, first_port: 0, last_port: 16383 } } }, OmicronZoneConfig { id: bd58e908-3308-4fab-8db4-ed67150a5a5d, underlay_address: fd00:1122:3344:101::3, zone_type: CockroachDb { address: [fd00:1122:3344:101::3]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, underlay_address: fd00:1122:3344:101::4, zone_type: CockroachDb { address: [fd00:1122:3344:101::4]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, underlay_address: fd00:1122:3344:101::5, zone_type: CockroachDb { address: [fd00:1122:3344:101::5]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: c999f854-acb5-47bf-95d4-4879391f2348, underlay_address: fd00:1122:3344:101::6, zone_type: CockroachDb { address: [fd00:1122:3344:101::6]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, underlay_address: fd00:1122:3344:101::7, zone_type: CockroachDb { address: [fd00:1122:3344:101::7]:32221, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: 8118d247-c8a4-47b6-be18-05eb663745b6, underlay_address: fd00:1122:3344:101::8, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } }, http_address: [fd00:1122:3344:101::8]:5353, dns_address: 10.151.2.164:53, nic: NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, underlay_address: fd00:1122:3344:101::9, zone_type: ExternalDns { dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } }, http_address: [fd00:1122:3344:101::9]:5353, dns_address: 10.151.2.165:53, nic: NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 } } }, OmicronZoneConfig { id: f3a19a9d-3117-4bf5-aa39-824f48040fae, underlay_address: fd00:1122:3344:101::a, zone_type: Nexus { internal_address: [fd00:1122:3344:101::a]:12221, external_ip: 10.151.2.166, nic: NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 62aaea40-4286-44d3-b143-827d830cadd5, underlay_address: fd00:1122:3344:101::b, zone_type: Nexus { internal_address: [fd00:1122:3344:101::b]:12221, external_ip: 10.151.2.167, nic: NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07, underlay_address: fd00:1122:3344:101::c, zone_type: Nexus { internal_address: [fd00:1122:3344:101::c]:12221, external_ip: 10.151.2.168, nic: NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, external_tls: true, external_dns_servers: [1.1.1.1, 9.9.9.9] } }, OmicronZoneConfig { id: 6021b400-4bb3-4adf-ac63-5c76d8b2bda6, underlay_address: fd00:1122:3344:101::d, zone_type: Oximeter { address: [fd00:1122:3344:101::d]:12223 } }, OmicronZoneConfig { id: 10b348d3-f952-455c-858e-798e7e59ab91, underlay_address: fd00:1122:3344:101::e, zone_type: Clickhouse { address: [fd00:1122:3344:101::e]:8123, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: ce51ce61-e883-4d12-b122-913a7a0e4a3b, underlay_address: fd00:1122:3344:101::f, zone_type: CruciblePantry { address: [fd00:1122:3344:101::f]:17000 } }, OmicronZoneConfig { id: ab7d0f8d-26db-4202-89ba-241139a6439a, underlay_address: fd00:1122:3344:101::10, zone_type: CruciblePantry { address: [fd00:1122:3344:101::10]:17000 } }, OmicronZoneConfig { id: 8c6fbfe9-c167-4d5c-bf93-ee11dc465b15, underlay_address: fd00:1122:3344:101::11, zone_type: CruciblePantry { address: [fd00:1122:3344:101::11]:17000 } }, OmicronZoneConfig { id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, underlay_address: fd00:1122:3344:101::12, zone_type: Crucible { address: [fd00:1122:3344:101::12]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External } } } }, OmicronZoneConfig { id: b88f007b-85df-4a7a-bde1-c68774a9ee57, underlay_address: fd00:1122:3344:101::13, zone_type: Crucible { address: [fd00:1122:3344:101::13]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External } } } }, OmicronZoneConfig { id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, underlay_address: fd00:1122:3344:101::14, zone_type: Crucible { address: [fd00:1122:3344:101::14]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External } } } }, OmicronZoneConfig { id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, underlay_address: fd00:1122:3344:101::15, zone_type: Crucible { address: [fd00:1122:3344:101::15]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External } } } }, OmicronZoneConfig { id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, underlay_address: fd00:1122:3344:101::16, zone_type: Crucible { address: [fd00:1122:3344:101::16]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External } } } }, OmicronZoneConfig { id: d84da9e5-ea99-4f13-a054-b6abd174b00d, underlay_address: fd00:1122:3344:101::17, zone_type: Crucible { address: [fd00:1122:3344:101::17]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External } } } }, OmicronZoneConfig { id: bd658f3d-7b61-4998-b6a3-589ca0776868, underlay_address: fd00:1122:3344:101::18, zone_type: Crucible { address: [fd00:1122:3344:101::18]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External } } } }, OmicronZoneConfig { id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, underlay_address: fd00:1122:3344:101::19, zone_type: Crucible { address: [fd00:1122:3344:101::19]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External } } } }, OmicronZoneConfig { id: 8e952779-16e7-4682-a009-9ce5d6fa3280, underlay_address: fd00:1122:3344:101::1a, zone_type: Crucible { address: [fd00:1122:3344:101::1a]:32345, dataset: OmicronZoneDataset { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External } } } }] } file = sled-agent/src/rack_setup/service.rs:431 sled_address = [fd00:1122:3344:101::1]:12345
29232024-06-21T21:40:57.748ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
29242024-06-21T21:40:57.748ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd6e5965-968d-4c3d-aa50-17b3d5747fdb, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: InternalDns }, responder: ... }
29252024-06-21T21:40:57.770ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
29262024-06-21T21:40:57.770ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6c75629f-c9c3-4cd0-8ac6-814984e78c82, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: InternalDns }, responder: ... }
29272024-06-21T21:40:57.787ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
29282024-06-21T21:40:57.787ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 33ad5a14-40c4-4b62-8de3-c1143a9861de, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: InternalDns }, responder: ... }
29292024-06-21T21:40:57.804ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
29302024-06-21T21:40:57.804ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd58e908-3308-4fab-8db4-ed67150a5a5d, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: CockroachDb }, responder: ... }
29312024-06-21T21:40:57.822ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
29322024-06-21T21:40:57.822ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 6ec51f86-b813-4d2c-bee0-1400d33a00d2, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
29332024-06-21T21:40:57.843ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
29342024-06-21T21:40:57.843ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9a9094ea-c1a1-4c5e-be6a-e79e1be407c9, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: CockroachDb }, responder: ... }
29352024-06-21T21:40:57.863ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
29362024-06-21T21:40:57.863ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: c999f854-acb5-47bf-95d4-4879391f2348, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: CockroachDb }, responder: ... }
29372024-06-21T21:40:57.884ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }) file = sled-storage/src/manager.rs:381
29382024-06-21T21:40:57.884ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 0aa1beae-e845-4690-9fd2-bf35912e37fc, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: CockroachDb }, responder: ... }
29392024-06-21T21:40:57.907ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
29402024-06-21T21:40:57.907ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8118d247-c8a4-47b6-be18-05eb663745b6, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: ExternalDns }, responder: ... }
29412024-06-21T21:40:57.929ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }) file = sled-storage/src/manager.rs:381
29422024-06-21T21:40:57.929ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: ExternalDns }, responder: ... }
29432024-06-21T21:40:57.953ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }) file = sled-storage/src/manager.rs:381
29442024-06-21T21:40:57.953ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 10b348d3-f952-455c-858e-798e7e59ab91, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Clickhouse }, responder: ... }
29452024-06-21T21:40:57.976ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29462024-06-21T21:40:57.976ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 9b262d50-7378-4dec-9a9e-7774b96af0c2, dataset_name: DatasetName { pool_name: ZpoolName { id: 5b452253-5f6d-4617-91b5-3f263edef77d (zpool), kind: External }, kind: Crucible }, responder: ... }
29472024-06-21T21:40:58.000ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29482024-06-21T21:40:58.000ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: b88f007b-85df-4a7a-bde1-c68774a9ee57, dataset_name: DatasetName { pool_name: ZpoolName { id: 41b82b1f-5def-467c-bc86-5c87195bd788 (zpool), kind: External }, kind: Crucible }, responder: ... }
29492024-06-21T21:40:58.024ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29502024-06-21T21:40:58.024ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8, dataset_name: DatasetName { pool_name: ZpoolName { id: 44b9c17b-6e37-45fe-b9da-6182ccbe59a4 (zpool), kind: External }, kind: Crucible }, responder: ... }
29512024-06-21T21:40:58.047ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29522024-06-21T21:40:58.047ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: a2a67eb9-3e4b-4f09-9ca5-64c505877ca6, dataset_name: DatasetName { pool_name: ZpoolName { id: e6f1c1a1-ab63-45ad-a43f-153910e0494a (zpool), kind: External }, kind: Crucible }, responder: ... }
29532024-06-21T21:40:58.069ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29542024-06-21T21:40:58.069ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 54f5f1b6-b399-4163-a9e3-0b7868c8be76, dataset_name: DatasetName { pool_name: ZpoolName { id: 8ce4d8fb-4017-46e3-ac55-3c40125a531b (zpool), kind: External }, kind: Crucible }, responder: ... }
29552024-06-21T21:40:58.090ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29562024-06-21T21:40:58.090ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: d84da9e5-ea99-4f13-a054-b6abd174b00d, dataset_name: DatasetName { pool_name: ZpoolName { id: a0c5f5f0-1cc5-45cc-9434-17b4abf3b972 (zpool), kind: External }, kind: Crucible }, responder: ... }
29572024-06-21T21:40:58.111ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29582024-06-21T21:40:58.111ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: bd658f3d-7b61-4998-b6a3-589ca0776868, dataset_name: DatasetName { pool_name: ZpoolName { id: 85a7fd4f-88d4-408c-a832-0d457152c72f (zpool), kind: External }, kind: Crucible }, responder: ... }
29592024-06-21T21:40:58.130ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29602024-06-21T21:40:58.130ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 1796d25b-1b93-4fd3-bf11-a40ad30fc679, dataset_name: DatasetName { pool_name: ZpoolName { id: 6b2e5449-891f-4dfd-8e2d-78189eabe443 (zpool), kind: External }, kind: Crucible }, responder: ... }
29612024-06-21T21:40:58.151ZINFOSledAgent (StorageManager): Received NewFilesystem(NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }) file = sled-storage/src/manager.rs:381
29622024-06-21T21:40:58.151ZINFOSledAgent (StorageManager): add_dataset file = sled-storage/src/manager.rs:809 request = NewFilesystemRequest { dataset_id: 8e952779-16e7-4682-a009-9ce5d6fa3280, dataset_name: DatasetName { pool_name: ZpoolName { id: 3ae04f9e-dd4d-4980-8e86-31c81824c907 (zpool), kind: External }, kind: Crucible }, responder: ... }
29632024-06-21T21:40:58.170ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29642024-06-21T21:40:58.170ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
29652024-06-21T21:40:58.170ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
29662024-06-21T21:40:58.170ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29672024-06-21T21:40:58.297ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
29682024-06-21T21:40:58.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29692024-06-21T21:40:58.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29702024-06-21T21:40:58.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29712024-06-21T21:40:58.297ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29722024-06-21T21:40:58.322ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
29732024-06-21T21:40:58.322ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29742024-06-21T21:40:58.322ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29752024-06-21T21:40:58.322ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29762024-06-21T21:40:58.334ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29772024-06-21T21:40:58.499ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
29782024-06-21T21:40:58.547ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
29792024-06-21T21:40:58.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29802024-06-21T21:40:58.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29812024-06-21T21:40:58.548ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29822024-06-21T21:40:58.550ZINFOSledAgent (ServiceManager): Resolving switch zone addresses in DNS file = sled-agent/src/bootstrap/early_networking.rs:263
29832024-06-21T21:40:58.550ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29842024-06-21T21:40:58.564ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29852024-06-21T21:40:58.564ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
29862024-06-21T21:40:58.564ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57 file = illumos-utils/src/zone.rs:315
29872024-06-21T21:40:58.570ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76 file = illumos-utils/src/zone.rs:315
29882024-06-21T21:40:58.575ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280 file = illumos-utils/src/zone.rs:315
29892024-06-21T21:40:58.580ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6 file = illumos-utils/src/zone.rs:315
29902024-06-21T21:40:58.586ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91 file = illumos-utils/src/zone.rs:315
29912024-06-21T21:40:58.591ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a file = illumos-utils/src/zone.rs:315
29922024-06-21T21:40:58.597ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6 file = illumos-utils/src/zone.rs:315
29932024-06-21T21:40:58.603ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868 file = illumos-utils/src/zone.rs:315
29942024-06-21T21:40:58.740ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57 file = illumos-utils/src/zone.rs:354
29952024-06-21T21:40:58.745ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76 file = illumos-utils/src/zone.rs:354
29962024-06-21T21:40:58.750ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280 file = illumos-utils/src/zone.rs:354
29972024-06-21T21:40:58.755ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6 file = illumos-utils/src/zone.rs:354
29982024-06-21T21:40:58.761ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91 file = illumos-utils/src/zone.rs:354
29992024-06-21T21:40:58.766ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a file = illumos-utils/src/zone.rs:354
30002024-06-21T21:40:58.771ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6 file = illumos-utils/src/zone.rs:354
30012024-06-21T21:40:58.777ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868 file = illumos-utils/src/zone.rs:354
30022024-06-21T21:40:58.784ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8 file = illumos-utils/src/zone.rs:315
30032024-06-21T21:40:58.791ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d file = illumos-utils/src/zone.rs:315
30042024-06-21T21:40:58.797ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679 file = illumos-utils/src/zone.rs:315
30052024-06-21T21:40:58.804ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15 file = illumos-utils/src/zone.rs:315
30062024-06-21T21:40:58.811ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b file = illumos-utils/src/zone.rs:315
30072024-06-21T21:40:58.817ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2 file = illumos-utils/src/zone.rs:315
30082024-06-21T21:40:58.825ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8 file = illumos-utils/src/zone.rs:354
30092024-06-21T21:40:58.832ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d file = illumos-utils/src/zone.rs:354
30102024-06-21T21:40:58.839ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679 file = illumos-utils/src/zone.rs:354
30112024-06-21T21:40:58.847ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15 file = illumos-utils/src/zone.rs:354
30122024-06-21T21:40:58.853ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b file = illumos-utils/src/zone.rs:354
30132024-06-21T21:40:58.859ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2 file = illumos-utils/src/zone.rs:354
30142024-06-21T21:40:58.931ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
30152024-06-21T21:40:58.994ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
30162024-06-21T21:40:59.075ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
30172024-06-21T21:40:59.128ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
30182024-06-21T21:40:59.194ZINFOSledAgent (ServiceManager): Querying MGS to determine switch location addr = fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:280
30192024-06-21T21:40:59.196ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
30202024-06-21T21:40:59.196ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
30212024-06-21T21:40:59.375ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte1", ip: 172.30.2.7, mac: MacAddr6([168, 64, 37, 255, 255, 165]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte1" } }
30222024-06-21T21:40:59.375ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1190 zone_type = nexus
30232024-06-21T21:40:59.380ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
30242024-06-21T21:40:59.380ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
30252024-06-21T21:40:59.495ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte2", ip: 172.30.2.6, mac: MacAddr6([168, 64, 37, 255, 159, 118]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte2" } }
30262024-06-21T21:40:59.495ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1190 zone_type = nexus
30272024-06-21T21:40:59.500ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
30282024-06-21T21:40:59.500ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
30292024-06-21T21:41:00.110ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte3", ip: 172.30.2.5, mac: MacAddr6([168, 64, 37, 255, 193, 73]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.2.1 }, vnic: "vopte3" } }
30302024-06-21T21:41:00.110ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1190 zone_type = nexus
30312024-06-21T21:41:00.110ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
30322024-06-21T21:41:00.110ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
30332024-06-21T21:41:00.110ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
30342024-06-21T21:41:00.110ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
30352024-06-21T21:41:00.142ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
30362024-06-21T21:41:00.142ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
30372024-06-21T21:41:00.588ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte4", ip: 172.30.1.6, mac: MacAddr6([168, 64, 37, 255, 153, 223]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte4" } }
30382024-06-21T21:41:00.588ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1190 zone_type = external_dns
30392024-06-21T21:41:00.592ZINFOSledAgent (ServiceManager): Found Switch0 at fd00:1122:3344:101::2 file = sled-agent/src/bootstrap/early_networking.rs:305
30402024-06-21T21:41:00.592ZINFOSledAgent (ServiceManager): Successfully looked up all expected switch zone underlay addresses addrs = {Switch0: fd00:1122:3344:101::2} file = sled-agent/src/bootstrap/early_networking.rs:197
30412024-06-21T21:41:00.743ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte5", ip: 172.30.1.5, mac: MacAddr6([168, 64, 37, 255, 241, 8]), slot: 0, vni: Vni { inner: 100 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.1.1 }, vnic: "vopte5" } }
30422024-06-21T21:41:00.743ZINFOSledAgent (ServiceManager): creating NAT entry for service file = sled-agent/src/services.rs:1190 zone_type = external_dns
30432024-06-21T21:41:00.744ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.168, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
30442024-06-21T21:41:00.754ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.167, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
30452024-06-21T21:41:00.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
30462024-06-21T21:41:00.758ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
30472024-06-21T21:41:01.682ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
30482024-06-21T21:41:01.682ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
30492024-06-21T21:41:01.683ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07 file = illumos-utils/src/zone.rs:315
30502024-06-21T21:41:01.733ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
30512024-06-21T21:41:01.733ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
30522024-06-21T21:41:01.734ZINFOSledAgent (ServiceManager): no nat entry found for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:102
30532024-06-21T21:41:01.734ZINFOSledAgent (ServiceManager): creating nat entry for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:109
30542024-06-21T21:41:01.734ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.166, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
30552024-06-21T21:41:01.759ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
30562024-06-21T21:41:01.760ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5 file = illumos-utils/src/zone.rs:315
30572024-06-21T21:41:01.785ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.165, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
30582024-06-21T21:41:01.791ZINFOSledAgent (ServiceManager): creation of nat entry successful for: V4( Ipv4Network { addr: 10.151.2.164, prefix: 32, }, ) file = clients/dpd-client/src/lib.rs:137
30592024-06-21T21:41:01.793ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
30602024-06-21T21:41:01.793ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
30612024-06-21T21:41:02.700ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07 file = illumos-utils/src/zone.rs:354
30622024-06-21T21:41:02.716ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5 file = illumos-utils/src/zone.rs:354
30632024-06-21T21:41:09.156ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae file = illumos-utils/src/zone.rs:315
30642024-06-21T21:41:09.177ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService10"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::13"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_41b82b1f-5def-467c-bc86-5c87195bd788/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::13"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="3764a982-4f70-490f-ad87-87dcf4d3f719"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30652024-06-21T21:41:09.180ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService11"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::16"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_8ce4d8fb-4017-46e3-ac55-3c40125a531b/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::16"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="e803a5a2-b132-41e4-87da-b4b830aff7bc"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30662024-06-21T21:41:09.184ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService12"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::1a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_3ae04f9e-dd4d-4980-8e86-31c81824c907/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1a"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="3b8f9f62-dbec-4906-acf9-645979269312"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30672024-06-21T21:41:09.188ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService17"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::18"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_85a7fd4f-88d4-408c-a832-0d457152c72f/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::18"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="bfd052dd-d63f-4187-8ef3-bcc7a413d5c6"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30682024-06-21T21:41:09.189ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf file = illumos-utils/src/zone.rs:315
30692024-06-21T21:41:09.203ZINFOSledAgent (ServiceManager): Configuring new Omicron zone: oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6 file = illumos-utils/src/zone.rs:315
30702024-06-21T21:41:09.224ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
30712024-06-21T21:41:09.257ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
30722024-06-21T21:41:09.267ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
30732024-06-21T21:41:09.292ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
30742024-06-21T21:41:09.301ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf file = illumos-utils/src/zone.rs:354
30752024-06-21T21:41:09.309ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6 file = illumos-utils/src/zone.rs:354
30762024-06-21T21:41:09.321ZINFOSledAgent (ServiceManager): Installing Omicron zone: oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae file = illumos-utils/src/zone.rs:354
30772024-06-21T21:41:09.456ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:53788
30782024-06-21T21:41:09.462ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 4751 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:53788 req_id = 77db912a-8b08-409c-a2a6-09293d72c9dc response_code = 200 uri = /baseboard
30792024-06-21T21:41:09.657ZINFOSledAgent (ServiceManager): Profile for oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService13"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::d"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/oximeter"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="id" value="6021b400-4bb3-4adf-ac63-5c76d8b2bda6"/> <propval type="astring" name="address" value="[fd00:1122:3344:101::d]:12223"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30802024-06-21T21:41:09.658ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
30812024-06-21T21:41:09.668ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService18"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::14"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_44b9c17b-6e37-45fe-b9da-6182ccbe59a4/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::14"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="b6dd3c49-ae5c-4543-b0d6-64a5faef6f02"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30822024-06-21T21:41:09.668ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
30832024-06-21T21:41:09.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 53.271583ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
30842024-06-21T21:41:09.832ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 47.340087ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
30852024-06-21T21:41:09.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 30.154829ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
30862024-06-21T21:41:09.871ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 62.912876ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
30872024-06-21T21:41:09.891ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 121.72442ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
30882024-06-21T21:41:09.921ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 146.90107ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
30892024-06-21T21:41:09.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 94.997065ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
30902024-06-21T21:41:09.990ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 59.498054ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
30912024-06-21T21:41:10.050ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 243.659852ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
30922024-06-21T21:41:10.082ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 244.853178ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
30932024-06-21T21:41:10.110ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 290.439848ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
30942024-06-21T21:41:10.134ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 120.101204ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
30952024-06-21T21:41:10.153ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 57.727264ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
30962024-06-21T21:41:10.154ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService16"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::15"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_e6f1c1a1-ab63-45ad-a43f-153910e0494a/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::15"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="f120cdcc-d3f1-4b24-b135-85a49151312f"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
30972024-06-21T21:41:10.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 71.357254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
30982024-06-21T21:41:10.174ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
30992024-06-21T21:41:10.237ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 93.911853ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31002024-06-21T21:41:10.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 117.058171ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31012024-06-21T21:41:10.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 558.633695ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
31022024-06-21T21:41:10.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 373.2347ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
31032024-06-21T21:41:10.829ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 461.137209ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
31042024-06-21T21:41:10.849ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 270.456646ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31052024-06-21T21:41:10.868ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 325.617954ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
31062024-06-21T21:41:10.897ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 179.94806ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31072024-06-21T21:41:10.901ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService19"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::17"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_a0c5f5f0-1cc5-45cc-9434-17b4abf3b972/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::17"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="d75b70bc-a858-48d9-a973-325d1811db4e"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31082024-06-21T21:41:10.909ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService20"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::19"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_6b2e5449-891f-4dfd-8e2d-78189eabe443/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::19"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="18443a44-52bd-403e-a0ed-658ce127c2c2"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31092024-06-21T21:41:10.909ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31102024-06-21T21:41:10.925ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31112024-06-21T21:41:10.935ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService21"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::11"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::11"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31122024-06-21T21:41:10.935ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31132024-06-21T21:41:11.081ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 40.709753ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31142024-06-21T21:41:11.102ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 573.843402ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
31152024-06-21T21:41:11.135ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 394.327605ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31162024-06-21T21:41:11.161ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 423.487376ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31172024-06-21T21:41:11.244ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 110.146839ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31182024-06-21T21:41:11.261ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 499.167627ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
31192024-06-21T21:41:11.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 867.68131ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
31202024-06-21T21:41:11.307ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 1.021419719s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
31212024-06-21T21:41:11.360ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 43.918409ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31222024-06-21T21:41:11.380ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 199.072404ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31232024-06-21T21:41:11.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 56.398999ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31242024-06-21T21:41:11.412ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 56.064936ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31252024-06-21T21:41:11.429ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 148.440498ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31262024-06-21T21:41:11.634ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 105.000403ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31272024-06-21T21:41:11.653ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 912.622419ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31282024-06-21T21:41:11.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 362.642892ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31292024-06-21T21:41:11.717ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 826.578952ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31302024-06-21T21:41:11.747ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 141.069031ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31312024-06-21T21:41:11.774ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 296.032196ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31322024-06-21T21:41:11.805ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 1.028119559s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
31332024-06-21T21:41:11.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 796.117912ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
31342024-06-21T21:41:11.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 231.061552ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31352024-06-21T21:41:11.914ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 216.607366ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31362024-06-21T21:41:12.063ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 1.184793821s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31372024-06-21T21:41:12.088ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 238.210785ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31382024-06-21T21:41:12.112ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 512.232113ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31392024-06-21T21:41:12.145ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 365.798054ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31402024-06-21T21:41:12.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 577.620461ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
31412024-06-21T21:41:12.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 559.130954ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31422024-06-21T21:41:12.357ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 1.131781645s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
31432024-06-21T21:41:12.532ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 1.113943543s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31442024-06-21T21:41:12.568ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 1.04821797s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31452024-06-21T21:41:12.588ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 957.805549ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31462024-06-21T21:41:12.641ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 707.910587ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31472024-06-21T21:41:12.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 800.270654ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
31482024-06-21T21:41:12.753ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 1.477863575s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
31492024-06-21T21:41:12.851ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 1.184849248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
31502024-06-21T21:41:12.852ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService27"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::9"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte4"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::9]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.6:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31512024-06-21T21:41:12.855ZINFOSledAgent (ServiceManager): Profile for oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService28"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::8"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte5"/> <propval type="astring" name="gateway" value="172.30.1.1"/> <propval type="astring" name="ip" value="172.30.1.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/external_dns"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="http_address" value="[fd00:1122:3344:101::8]:5353"/> <propval type="astring" name="dns_address" value="172.30.1.5:53"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31522024-06-21T21:41:12.855ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31532024-06-21T21:41:12.864ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31542024-06-21T21:41:12.919ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 1.404116877s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31552024-06-21T21:41:13.146ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 74.235545ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31562024-06-21T21:41:13.170ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 29.214011ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31572024-06-21T21:41:13.215ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 127.276593ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31582024-06-21T21:41:13.236ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 131.410748ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31592024-06-21T21:41:13.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 1.02241259s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31602024-06-21T21:41:13.374ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 253.051103ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31612024-06-21T21:41:13.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 244.660547ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31622024-06-21T21:41:13.450ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 668.023224ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31632024-06-21T21:41:13.478ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 763.397433ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
31642024-06-21T21:41:13.506ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 868.344843ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
31652024-06-21T21:41:13.567ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 811.527275ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31662024-06-21T21:41:13.637ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 1.349155633s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31672024-06-21T21:41:13.687ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 576.135666ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31682024-06-21T21:41:13.752ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 465.402784ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31692024-06-21T21:41:13.778ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 1.340038222s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31702024-06-21T21:41:14.051ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 621.512167ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
31712024-06-21T21:41:14.136ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 1.241112548s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
31722024-06-21T21:41:14.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 449.642759ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31732024-06-21T21:41:14.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 907.842084ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
31742024-06-21T21:41:14.255ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 1.08376848s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
31752024-06-21T21:41:14.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 421.349806ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31762024-06-21T21:41:14.339ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 1.212246616s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
31772024-06-21T21:41:14.354ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 1.288380001s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
31782024-06-21T21:41:14.387ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 680.965092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
31792024-06-21T21:41:14.396ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 1.485705115s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
31802024-06-21T21:41:14.666ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService26"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::a"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte3"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.5"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31812024-06-21T21:41:14.667ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
31822024-06-21T21:41:14.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 967.686465ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
31832024-06-21T21:41:14.683ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService24"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::c"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte1"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.7"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31842024-06-21T21:41:14.684ZINFOSledAgent (ServiceManager): Profile for oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService25"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::b"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="oxide/opte-interface-setup"> <property_group type="application" name="config"> <propval type="astring" name="interface" value="vopte2"/> <propval type="astring" name="gateway" value="172.30.2.1"/> <propval type="astring" name="ip" value="172.30.2.6"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/nexus"> <instance enabled="true" name="default"> <property_group type="application" name="config"> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
31852024-06-21T21:41:14.693ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 922.897663ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
31862024-06-21T21:41:14.693ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
31872024-06-21T21:41:14.698ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
31882024-06-21T21:41:14.713ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 1.300746576s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
31892024-06-21T21:41:14.865ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 64.58339ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
31902024-06-21T21:41:15.012ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 31.371889ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
31912024-06-21T21:41:15.100ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 1.144746669s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
31922024-06-21T21:41:15.119ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 97.19211ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
31932024-06-21T21:41:15.139ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 1.022355724s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
31942024-06-21T21:41:15.159ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 54.488383ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
31952024-06-21T21:41:15.184ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 58.795393ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
31962024-06-21T21:41:15.208ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 520.531589ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
31972024-06-21T21:41:15.239ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 772.661996ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
31982024-06-21T21:41:15.265ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 291.687011ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
31992024-06-21T21:41:15.285ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 73.576904ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32002024-06-21T21:41:15.305ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 253.281418ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
32012024-06-21T21:41:15.362ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 1.345561751s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
32022024-06-21T21:41:15.384ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 273.262336ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32032024-06-21T21:41:15.399ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 660.810061ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
32042024-06-21T21:41:15.576ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 1.042109378s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
32052024-06-21T21:41:15.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 325.199391ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
32062024-06-21T21:41:15.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 549.781414ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
32072024-06-21T21:41:15.630ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 1.456240882s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
32082024-06-21T21:41:15.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 981.616391ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
32092024-06-21T21:41:15.671ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 713.089576ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
32102024-06-21T21:41:15.683ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 376.135766ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32112024-06-21T21:41:15.742ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 684.349813ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
32122024-06-21T21:41:15.895ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 954.3376ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
32132024-06-21T21:41:15.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 973.951263ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
32142024-06-21T21:41:16.025ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 1.105932108s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
32152024-06-21T21:41:16.036ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 1.178098786s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
32162024-06-21T21:41:16.073ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 510.449407ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32172024-06-21T21:41:16.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 807.465045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
32182024-06-21T21:41:16.145ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService15"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::10"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::10"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
32192024-06-21T21:41:16.147ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32202024-06-21T21:41:16.157ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService23"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::12"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/agent"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="dataset" value="oxp_5b452253-5f6d-4617-91b5-3f263edef77d/crucible"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::12"/> <propval type="astring" name="listen_port" value="32345"/> <propval type="astring" name="uuid" value="89bcb856-d840-41ca-b8ed-c21fd995ec45"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
32212024-06-21T21:41:16.175ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 1.119557429s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
32222024-06-21T21:41:16.175ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32232024-06-21T21:41:16.200ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 1.287879248s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
32242024-06-21T21:41:16.200ZINFOSledAgent (ServiceManager): Profile for oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService22"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::f"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/crucible/pantry"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::f"/> <propval type="astring" name="listen_port" value="17000"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
32252024-06-21T21:41:16.202ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32262024-06-21T21:41:16.258ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 790.598254ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
32272024-06-21T21:41:16.402ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 831.459462ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
32282024-06-21T21:41:16.426ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 38.169432ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32292024-06-21T21:41:16.446ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 1.310703478s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
32302024-06-21T21:41:16.472ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 35.696899ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32312024-06-21T21:41:16.493ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 83.293256ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32322024-06-21T21:41:16.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 25.677731ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32332024-06-21T21:41:16.538ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 144.584045ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32342024-06-21T21:41:16.655ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 141.892458ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32352024-06-21T21:41:16.675ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 240.534065ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32362024-06-21T21:41:16.694ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 534.827392ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
32372024-06-21T21:41:16.706ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 580.359814ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32382024-06-21T21:41:16.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 891.342252ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
32392024-06-21T21:41:16.728ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 269.665022ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32402024-06-21T21:41:16.738ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 1.08962326s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
32412024-06-21T21:41:16.818ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 196.675653ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32422024-06-21T21:41:16.869ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 572.116441ms file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
32432024-06-21T21:41:16.917ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 503.508823ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
32442024-06-21T21:41:16.935ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 1.015494313s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
32452024-06-21T21:41:16.957ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 429.985023ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32462024-06-21T21:41:17.015ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 269.14761ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32472024-06-21T21:41:17.029ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 412.187113ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32482024-06-21T21:41:17.065ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 550.982229ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
32492024-06-21T21:41:17.108ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 1.0488689s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
32502024-06-21T21:41:17.154ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 747.321792ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
32512024-06-21T21:41:17.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 1.443398164s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
32522024-06-21T21:41:17.254ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 602.491644ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
32532024-06-21T21:41:17.272ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 916.065497ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
32542024-06-21T21:41:17.306ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 456.88363ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32552024-06-21T21:41:17.322ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 704.949845ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32562024-06-21T21:41:17.340ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 963.875266ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
32572024-06-21T21:41:17.407ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 842.890573ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32582024-06-21T21:41:17.441ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 1.425666676s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
32592024-06-21T21:41:17.463ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 427.039546ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32602024-06-21T21:41:17.481ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 1.150435091s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
32612024-06-21T21:41:17.485ZINFOSledAgent (ServiceManager): Profile for oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="oxide/zone-network-setup"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlService14"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="static_addr" value="fd00:1122:3344:101::e"/> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/ssh"> <instance enabled="false" name="default"> </instance> </service> <service version="1" type="service" name="oxide/clickhouse"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::e"/> <propval type="astring" name="listen_port" value="8123"/> <propval type="astring" name="store" value="/data"/> </property_group> </instance> </service> <service version="1" type="service" name="network/dns/install"> <property_group type="application" name="install_props"> <property type="net_address" name="nameserver"> <net_address_list> <value_node value="fd00:1122:3344:3::1" /> <value_node value="fd00:1122:3344:2::1" /> <value_node value="fd00:1122:3344:1::1" /> </net_address_list> </property> </property_group> <instance enabled="true" name="default"> </instance> </service> <service version="1" type="service" name="network/dns/client"> <instance enabled="true" name="default"> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34
32622024-06-21T21:41:17.486ZINFOSledAgent (ServiceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
32632024-06-21T21:41:17.510ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 1.249932277s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
32642024-06-21T21:41:17.631ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 1.363285496s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
32652024-06-21T21:41:17.647ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 673.75777ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
32662024-06-21T21:41:17.764ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 34.364584ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
32672024-06-21T21:41:17.781ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 1.299512537s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
32682024-06-21T21:41:17.795ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 722.158652ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32692024-06-21T21:41:17.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 74.870847ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
32702024-06-21T21:41:17.851ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 563.964786ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
32712024-06-21T21:41:17.925ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 1.335589435s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
32722024-06-21T21:41:17.948ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 1.495439457s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
32732024-06-21T21:41:17.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 237.91536ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
32742024-06-21T21:41:17.975ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 1.357924075s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32752024-06-21T21:41:17.988ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 1.156886647s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
32762024-06-21T21:41:18.046ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 741.764049ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32772024-06-21T21:41:18.174ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 1.055919374s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
32782024-06-21T21:41:18.214ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 1.150079414s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
32792024-06-21T21:41:18.230ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 410.496955ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
32802024-06-21T21:41:18.266ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 1.421135676s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
32812024-06-21T21:41:18.316ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 1.380046221s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
32822024-06-21T21:41:18.335ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 1.493488184s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
32832024-06-21T21:41:18.431ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 671.369726ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
32842024-06-21T21:41:18.531ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 1.172504561s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
32852024-06-21T21:41:18.644ZWARNSledAgent (ServiceManager): wait for service Some("oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6") failed: Property not found. retry in 1.074372358s file = illumos-utils/src/svc.rs:36 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
32862024-06-21T21:41:18.657ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 842.148857ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
32872024-06-21T21:41:18.692ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 525.501069ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
32882024-06-21T21:41:18.775ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280") failed: Property not found. retry in 898.049365ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
32892024-06-21T21:41:18.803ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 1.181289061s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
32902024-06-21T21:41:18.881ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 691.153783ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
32912024-06-21T21:41:19.008ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 1.102135176s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
32922024-06-21T21:41:19.096ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 809.722929ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
32932024-06-21T21:41:19.118ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868") failed: Property not found. retry in 799.048611ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
32942024-06-21T21:41:19.157ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 662.352677ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
32952024-06-21T21:41:19.232ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 1.464991597s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
32962024-06-21T21:41:19.245ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 990.803136ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
32972024-06-21T21:41:19.274ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 900.596369ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
32982024-06-21T21:41:19.352ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 1.153218635s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
32992024-06-21T21:41:19.379ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76") failed: Property not found. retry in 1.258596341s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
33002024-06-21T21:41:19.458ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57") failed: Property not found. retry in 1.247528805s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
33012024-06-21T21:41:19.515ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 535.570864ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33022024-06-21T21:41:19.590ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 1.362263201s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
33032024-06-21T21:41:19.718ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 1.107454544s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
33042024-06-21T21:41:19.733ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 1.207407931s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
33052024-06-21T21:41:19.751ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 839.633493ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33062024-06-21T21:41:19.840ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 516.353806ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33072024-06-21T21:41:19.856ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 820.114981ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
33082024-06-21T21:41:19.928ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 534.444339ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
33092024-06-21T21:41:20.005ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 786.717832ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
33102024-06-21T21:41:20.074ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 564.736116ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33112024-06-21T21:41:20.125ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 923.097976ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
33122024-06-21T21:41:20.187ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 1.435595738s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
33132024-06-21T21:41:20.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 985.803084ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
33142024-06-21T21:41:20.371ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 1.21500355s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33152024-06-21T21:41:20.480ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 714.096982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
33162024-06-21T21:41:20.524ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 973.100289ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
33172024-06-21T21:41:20.607ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 1.342270093s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33182024-06-21T21:41:20.672ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 734.210254ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33192024-06-21T21:41:20.688ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 1.389120715s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
33202024-06-21T21:41:20.709ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 776.441563ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
33212024-06-21T21:41:20.811ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 613.886481ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
33222024-06-21T21:41:20.841ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 1.287474554s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
33232024-06-21T21:41:20.960ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 1.12197908s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
33242024-06-21T21:41:20.976ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 1.075488832s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
33252024-06-21T21:41:21.066ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 569.690422ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
33262024-06-21T21:41:21.211ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 1.28767986s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
33272024-06-21T21:41:21.250ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 1.247408527s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
33282024-06-21T21:41:21.419ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 684.033063ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33292024-06-21T21:41:21.438ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 781.535813ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
33302024-06-21T21:41:21.501ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 762.174405ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
33312024-06-21T21:41:21.517ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 553.603773ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
33322024-06-21T21:41:21.605ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 826.782992ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33332024-06-21T21:41:21.638ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 1.339928927s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
33342024-06-21T21:41:21.653ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 946.202977ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
33352024-06-21T21:41:21.962ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 749.403091ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33362024-06-21T21:41:22.067ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 664.887153ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
33372024-06-21T21:41:22.084ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 1.247195101s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
33382024-06-21T21:41:22.098ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8") failed: Property not found. retry in 1.469634362s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
33392024-06-21T21:41:22.111ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 1.223657865s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
33402024-06-21T21:41:22.123ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 1.467569998s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33412024-06-21T21:41:22.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 1.043672134s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
33422024-06-21T21:41:22.234ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 1.342281782s file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
33432024-06-21T21:41:22.276ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 530.281198ms file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
33442024-06-21T21:41:22.443ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 685.735052ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33452024-06-21T21:41:22.512ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf") failed: Property not found. retry in 1.472757559s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
33462024-06-21T21:41:22.522ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 660.441092ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
33472024-06-21T21:41:22.612ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15") failed: Property not found. retry in 1.089628445s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
33482024-06-21T21:41:22.726ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 722.353804ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33492024-06-21T21:41:22.743ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d") failed: Property not found. retry in 970.476737ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
33502024-06-21T21:41:22.820ZWARNSledAgent (ServiceManager): wait for service Some("oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6") failed: Property not found. retry in 1.270951509s file = illumos-utils/src/svc.rs:36 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
33512024-06-21T21:41:22.991ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6") failed: Property not found. retry in 975.177464ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
33522024-06-21T21:41:23.141ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 793.9611ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33532024-06-21T21:41:23.198ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 1.318240647s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
33542024-06-21T21:41:23.209ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679") failed: Property not found. retry in 874.236837ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
33552024-06-21T21:41:23.343ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 1.253852016s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
33562024-06-21T21:41:23.353ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae") failed: Property not found. retry in 683.992357ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
33572024-06-21T21:41:23.460ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 634.122982ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33582024-06-21T21:41:23.597ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5") failed: Property not found. retry in 938.082654ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
33592024-06-21T21:41:23.608ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 915.230829ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33602024-06-21T21:41:23.952ZWARNSledAgent (ServiceManager): wait for service Some("oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07") failed: Property not found. retry in 827.073032ms file = illumos-utils/src/svc.rs:36 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33612024-06-21T21:41:24.122ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b") failed: Property not found. retry in 1.350824988s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33622024-06-21T21:41:24.533ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a") failed: Property not found. retry in 860.422657ms file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
33632024-06-21T21:41:24.551ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 1.246424126s file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33642024-06-21T21:41:24.609ZWARNSledAgent (ServiceManager): wait for service Some("oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2") failed: Property not found. retry in 1.156522603s file = illumos-utils/src/svc.rs:36 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
33652024-06-21T21:41:25.813ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 731.348755ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33662024-06-21T21:41:26.563ZWARNSledAgent (ServiceManager): wait for service Some("oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91") failed: Property not found. retry in 768.273189ms file = illumos-utils/src/svc.rs:36 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33672024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_b88f007b-85df-4a7a-bde1-c68774a9ee57
33682024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_54f5f1b6-b399-4163-a9e3-0b7868c8be76
33692024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_8e952779-16e7-4682-a009-9ce5d6fa3280
33702024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_oximeter_6021b400-4bb3-4adf-ac63-5c76d8b2bda6
33712024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_nexus_3bdb0a0f-906b-4629-8c23-bd2bba1cce07
33722024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_clickhouse_10b348d3-f952-455c-858e-798e7e59ab91
33732024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_pantry_ab7d0f8d-26db-4202-89ba-241139a6439a
33742024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_a2a67eb9-3e4b-4f09-9ca5-64c505877ca6
33752024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_nexus_62aaea40-4286-44d3-b143-827d830cadd5
33762024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_bd658f3d-7b61-4998-b6a3-589ca0776868
33772024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_external_dns_1acd5e4e-4ad7-4431-b537-a28da2f217bf
33782024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_a7a0de82-3bf3-4ebb-89a5-e8dc8d06cbe8
33792024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_d84da9e5-ea99-4f13-a054-b6abd174b00d
33802024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_1796d25b-1b93-4fd3-bf11-a40ad30fc679
33812024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_external_dns_8118d247-c8a4-47b6-be18-05eb663745b6
33822024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_pantry_8c6fbfe9-c167-4d5c-bf93-ee11dc465b15
33832024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_nexus_f3a19a9d-3117-4bf5-aa39-824f48040fae
33842024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_pantry_ce51ce61-e883-4d12-b122-913a7a0e4a3b
33852024-06-21T21:41:27.363ZINFOSledAgent (ServiceManager): Zone started file = sled-agent/src/services.rs:2956 zone = oxz_crucible_9b262d50-7378-4dec-9a9e-7774b96af0c2
33862024-06-21T21:41:27.364ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33872024-06-21T21:41:27.368ZINFOSledAgent (ServiceManager): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.omicron-zones.json.tmp file = common/src/ledger.rs:199
33882024-06-21T21:41:27.374ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 29629653 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::1]:33425 req_id = 2a1ca099-b3db-415a-8636-77d56da69b37 response_code = 204 uri = /omicron-zones
33892024-06-21T21:41:27.406ZINFOSledAgent (RSS): Finished setting up services file = sled-agent/src/rack_setup/service.rs:1244
33902024-06-21T21:41:27.414ZINFOSledAgent (RSS): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
33912024-06-21T21:41:27.414ZINFOSledAgent (RSS): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/.rss-plan-completed.marker.tmp file = common/src/ledger.rs:199
33922024-06-21T21:41:27.416ZINFOSledAgent (RSS): Handing off control to Nexus file = sled-agent/src/rack_setup/service.rs:681
33932024-06-21T21:41:27.419ZINFOSledAgent (RSS): Nexus address: [fd00:1122:3344:101::c]:12221 file = sled-agent/src/rack_setup/service.rs:694
33942024-06-21T21:41:27.465ZINFOSledAgent (RSS): rack_network_config: RackNetworkConfigV2 { bfd: [], bgp: [], infra_ip_first: 10.151.2.174, infra_ip_last: 10.151.2.174, ports: [ PortConfigV2 { addresses: [ UplinkAddressConfig { address: V4( Ipv4Net { addr: 10.151.2.174, width: 24, }, ), vlan_id: None, }, ], autoneg: false, bgp_peers: [], port: "qsfp0", routes: [ RouteConfig { destination: V4( Ipv4Net { addr: 0.0.0.0, width: 0, }, ), nexthop: 10.151.2.100, vlan_id: None, }, ], switch: Switch0, uplink_port_fec: None, uplink_port_speed: Speed40G, }, ], rack_subnet: Ipv6Net { addr: fd00:1122:3344:100::, width: 56, }, } file = sled-agent/src/rack_setup/service.rs:825
33952024-06-21T21:41:27.473ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
33962024-06-21T21:41:27.624ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
33972024-06-21T21:41:27.703ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
33982024-06-21T21:41:27.895ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
33992024-06-21T21:41:28.083ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34002024-06-21T21:41:28.241ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34012024-06-21T21:41:28.483ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34022024-06-21T21:41:28.671ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34032024-06-21T21:41:28.857ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34042024-06-21T21:41:29.471ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34052024-06-21T21:41:30.011ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34062024-06-21T21:41:30.447ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34072024-06-21T21:41:31.507ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34082024-06-21T21:41:32.026ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34092024-06-21T21:41:32.333ZWARNSledAgent (producer-registration-task): failed to register as a producer with Nexus, will retry delay = 158.655787672s error = "Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/metrics/producers): error trying to connect: tcp connect error: Connection refused (os error 146)" file = oximeter/producer/src/lib.rs:417 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
34102024-06-21T21:41:32.976ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34112024-06-21T21:41:34.459ZINFOSledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:1122:3344:101::c]:12221/racks/884d2112-d3cd-48eb-9cc7-ecb0a29f78a9/initialization-complete): error trying to connect: tcp connect error: Connection refused (os error 146) file = sled-agent/src/rack_setup/service.rs:887
34122024-06-21T21:41:37.297ZINFOSledAgent (NexusNotifierTask): Successfully put SledAgentInfo to nexus file = sled-agent/src/nexus.rs:592 new_generation = 1 old_generation = 1 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
34132024-06-21T21:41:37.434ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48849
34142024-06-21T21:41:37.436ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34152024-06-21T21:41:37.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 86138 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48849 req_id = a981d186-cb30-47f4-8569-abaa4f2145f6 response_code = 200 uri = /inventory
34162024-06-21T21:41:37.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49044
34172024-06-21T21:41:37.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:34256
34182024-06-21T21:41:37.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63481
34192024-06-21T21:41:37.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42213
34202024-06-21T21:41:37.521ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34212024-06-21T21:41:37.531ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34222024-06-21T21:41:37.531ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34232024-06-21T21:41:37.531ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
34242024-06-21T21:41:37.546ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:50803
34252024-06-21T21:41:37.555ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 9605 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:50803 req_id = 7ea4bb5d-7537-48be-a132-fdb3df456d7e response_code = 200 uri = /v2p
34262024-06-21T21:41:37.580ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
34272024-06-21T21:41:37.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 59393 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:34256 req_id = e9588e67-c6ce-474b-97ec-8c2a03993471 response_code = 200 uri = /v2p
34282024-06-21T21:41:37.581ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 51159 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49044 req_id = 6722a371-1476-4791-8a5e-41d29ad12569 response_code = 200 uri = /v2p
34292024-06-21T21:41:37.582ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 51671 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48849 req_id = 244415c5-96de-4c7c-90ca-7b1ec0628a18 response_code = 200 uri = /omicron-zones
34302024-06-21T21:41:37.664ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 142552 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63481 req_id = a5f1facd-4785-43b1-bbfe-4ddc483c3313 response_code = 200 uri = /inventory
34312024-06-21T21:41:37.664ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34322024-06-21T21:41:37.664ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
34332024-06-21T21:41:37.665ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
34342024-06-21T21:41:37.665ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 988 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63481 req_id = baa924ac-132e-4b8a-82d6-898b9dbeb26a response_code = 200 uri = /omicron-zones
34352024-06-21T21:41:37.670ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 148726 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42213 req_id = a28c17b9-c45c-4b33-acf3-11370bea9cfd response_code = 200 uri = /inventory
34362024-06-21T21:41:37.670ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34372024-06-21T21:41:37.670ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
34382024-06-21T21:41:37.670ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
34392024-06-21T21:41:37.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 19004 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42213 req_id = fb24eb5f-0ceb-4d76-973f-62e2594c9296 response_code = 200 uri = /omicron-zones
34402024-06-21T21:41:39.438ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61461
34412024-06-21T21:41:39.439ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34422024-06-21T21:41:39.533ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 94598 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61461 req_id = e00f0796-8073-41ae-b60f-125f15885c14 response_code = 200 uri = /inventory
34432024-06-21T21:41:39.533ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33506
34442024-06-21T21:41:39.534ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34452024-06-21T21:41:39.534ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
34462024-06-21T21:41:39.534ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 447 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33506 req_id = 187b44fa-7712-4152-a99e-c6af4867231a response_code = 200 uri = /baseboard
34472024-06-21T21:41:39.534ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
34482024-06-21T21:41:39.534ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 966 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61461 req_id = 8de69d19-6845-4ebe-9a57-7a341e9660d5 response_code = 200 uri = /omicron-zones
34492024-06-21T21:41:39.677ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:35671
34502024-06-21T21:41:39.703ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34512024-06-21T21:41:39.703ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34522024-06-21T21:41:39.716ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34532024-06-21T21:41:39.716ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34542024-06-21T21:41:39.717ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34552024-06-21T21:41:39.717ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
34562024-06-21T21:41:39.717ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34572024-06-21T21:41:39.717ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 40240 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:35671 req_id = 3ba599af-c0d3-4e89-b09c-0eebce210ea1 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34582024-06-21T21:41:39.719ZINFOSledAgent (RSS): Handoff to Nexus is complete file = sled-agent/src/rack_setup/service.rs:897
34592024-06-21T21:41:39.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45306
34602024-06-21T21:41:39.767ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57296
34612024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34622024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34632024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34642024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34652024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34662024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
34672024-06-21T21:41:39.768ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34682024-06-21T21:41:39.768ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 324 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:45306 req_id = 1b51952f-373f-404c-9933-244cb3770a49 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34692024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34702024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34712024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34722024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34732024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34742024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
34752024-06-21T21:41:39.769ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34762024-06-21T21:41:39.769ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 837 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:57296 req_id = d3bf0cfb-8f4f-45b3-9f0b-73bdf7206960 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34772024-06-21T21:41:39.923ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:54285
34782024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
34792024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34802024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34812024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
34822024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34832024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
34842024-06-21T21:41:39.923ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
34852024-06-21T21:41:39.923ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 328 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:54285 req_id = 9d690b26-b54e-44f6-a7ca-2a83fe070911 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
34862024-06-21T21:41:40.652ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:38679
34872024-06-21T21:41:40.652ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34882024-06-21T21:41:40.742ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 89754 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38679 req_id = a468229e-3a2d-4530-b769-0eaccabf3ae4 response_code = 200 uri = /inventory
34892024-06-21T21:41:40.742ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
34902024-06-21T21:41:40.742ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/config/omicron-zones.json file = common/src/ledger.rs:177
34912024-06-21T21:41:40.743ZINFOSledAgent (ServiceManager): Reading ledger from /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/config/omicron-zones.json file = common/src/ledger.rs:177
34922024-06-21T21:41:40.743ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1140 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:38679 req_id = 0e964566-294b-47eb-80f8-16953493fae0 response_code = 200 uri = /omicron-zones
34932024-06-21T21:42:05.493ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34120
34942024-06-21T21:42:05.493ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 193 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34120 req_id = 7f04797c-0944-48e4-a298-9c766faa6b3e response_code = 200 uri = /v2p
34952024-06-21T21:42:05.546ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:65492
34962024-06-21T21:42:05.547ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 118 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:65492 req_id = 47cb262b-c8cb-4ccf-a7d9-f2f1edc8bfb7 response_code = 200 uri = /v2p
34972024-06-21T21:42:05.576ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:46508
34982024-06-21T21:42:05.577ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 272 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:46508 req_id = 780442b0-457e-4716-b047-af9d0e7ff511 response_code = 200 uri = /v2p
34992024-06-21T21:42:05.592ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61609
35002024-06-21T21:42:05.678ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 85849 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:61609 req_id = 2b41e8b5-82b1-4950-9adf-88ae0aa8d5fc response_code = 204 uri = /switch-ports
35012024-06-21T21:42:05.678ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:53114
35022024-06-21T21:42:05.728ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 49566 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:53114 req_id = 30eb2cd1-3532-4cfd-80e6-16de087aac84 response_code = 204 uri = /switch-ports
35032024-06-21T21:42:05.728ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:35737
35042024-06-21T21:42:05.733ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 4907 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61609 req_id = 2890c30c-17d0-4b32-9ebc-66fd122bc822 response_code = 200 uri = /network-bootstore-config
35052024-06-21T21:42:05.733ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 4833 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:53114 req_id = 0f0bd8c4-7a3c-4422-af9f-a04f0c7a6c3a response_code = 200 uri = /network-bootstore-config
35062024-06-21T21:42:05.761ZINFOSledAgent (bootstore): Attempting to update network config with generation: 2, current_generation: 1 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
35072024-06-21T21:42:05.761ZINFOSledAgent (bootstore): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
35082024-06-21T21:42:05.769ZINFOSledAgent (bootstore): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
35092024-06-21T21:42:05.770ZINFOSledAgent (bootstore): Broadcasting network config with generation 2 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
35102024-06-21T21:42:05.770ZINFOSledAgent (bootstore): Attempting to update network config with generation: 3, current_generation: 2 file = bootstore/src/schemes/v0/peer.rs:564 peer_id = pc-sock-i86pc
35112024-06-21T21:42:05.770ZINFOSledAgent (bootstore): Writing ledger to /pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
35122024-06-21T21:42:05.770ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 11646 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61609 req_id = 0ebbe658-4ab8-4859-a5ee-8daf27f1d9c4 response_code = 204 uri = /network-bootstore-config
35132024-06-21T21:42:05.770ZINFOSledAgent (bootstore): Writing ledger to /pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/cluster/.bootstore-network-config.json.tmp file = common/src/ledger.rs:199 peer_id = pc-sock-i86pc
35142024-06-21T21:42:05.770ZINFOSledAgent (bootstore): Broadcasting network config with generation 3 file = bootstore/src/schemes/v0/peer.rs:648 peer_id = pc-sock-i86pc
35152024-06-21T21:42:05.771ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3458 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:53114 req_id = bab53288-3033-4eab-a2e4-0c56acf5a687 response_code = 204 uri = /network-bootstore-config
35162024-06-21T21:42:05.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47121 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:35737 req_id = 990b5b61-b661-42f2-a805-7fa19dc20a19 response_code = 204 uri = /switch-ports
35172024-06-21T21:42:05.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 87 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:35737 req_id = a16a3fc9-0c2d-4eeb-888c-de36ded1333c response_code = 200 uri = /network-bootstore-config
35182024-06-21T21:42:09.567ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:65269
35192024-06-21T21:42:09.568ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 89 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:65269 req_id = 19b31cdb-bba5-4731-a963-5a0914b27bc9 response_code = 200 uri = /baseboard
35202024-06-21T21:42:26.971ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
35212024-06-21T21:42:35.418ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52401
35222024-06-21T21:42:35.418ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 289 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52401 req_id = 58c564df-3b4f-4cfa-83e7-b0583e4830a2 response_code = 200 uri = /v2p
35232024-06-21T21:42:35.629ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65174
35242024-06-21T21:42:35.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 150 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:65174 req_id = a6a2113a-8669-4c55-a064-14e408e7c1da response_code = 200 uri = /v2p
35252024-06-21T21:42:35.850ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57999
35262024-06-21T21:42:35.850ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 274 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57999 req_id = 942d46c1-414c-4e0e-b8b8-6dba4844ce29 response_code = 200 uri = /v2p
35272024-06-21T21:42:35.858ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63405
35282024-06-21T21:42:35.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44837 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:63405 req_id = 9ff92ef1-0434-4520-ba4b-36a8354b5f42 response_code = 204 uri = /switch-ports
35292024-06-21T21:42:35.903ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 99 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63405 req_id = 34a0c151-0e83-4e2b-80a5-8a1046f9c185 response_code = 200 uri = /network-bootstore-config
35302024-06-21T21:42:35.925ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:58525
35312024-06-21T21:42:35.971ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46119 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:58525 req_id = 8aed77a8-e51b-4431-8ff5-1c4703794408 response_code = 204 uri = /switch-ports
35322024-06-21T21:42:35.972ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:58525 req_id = fad82a98-b40f-425c-87e0-524b79088ddb response_code = 200 uri = /network-bootstore-config
35332024-06-21T21:42:36.102ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38702
35342024-06-21T21:42:36.149ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46124 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:38702 req_id = bee38ec6-93b1-4816-9ae3-50bd1692890d response_code = 204 uri = /switch-ports
35352024-06-21T21:42:36.149ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38702 req_id = 683a491c-a9f3-4711-858c-2ec6e2f684eb response_code = 200 uri = /network-bootstore-config
35362024-06-21T21:42:39.601ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:58798
35372024-06-21T21:42:39.601ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 62 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:58798 req_id = 87ea77ab-08e1-4a1c-8e95-f3c161523ca0 response_code = 200 uri = /baseboard
35382024-06-21T21:43:05.313ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57161
35392024-06-21T21:43:05.313ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 190 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57161 req_id = e0b74ef4-3d07-4050-885e-e42501b97623 response_code = 200 uri = /v2p
35402024-06-21T21:43:05.421ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:64243
35412024-06-21T21:43:05.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 52702 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:64243 req_id = a57712ea-605e-468e-afe7-ffabce93ee91 response_code = 204 uri = /switch-ports
35422024-06-21T21:43:05.474ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 104 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:64243 req_id = e16ebba3-c68d-4a52-a203-1d6a4c43b151 response_code = 200 uri = /network-bootstore-config
35432024-06-21T21:43:05.521ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48797
35442024-06-21T21:43:05.521ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48797 req_id = b3ed5d6a-bd45-4c1c-a40b-8b2747b48e98 response_code = 200 uri = /v2p
35452024-06-21T21:43:05.536ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42357
35462024-06-21T21:43:05.536ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 82 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42357 req_id = 0c23e993-1774-4a62-af4c-a6cefd1e5e5d response_code = 200 uri = /v2p
35472024-06-21T21:43:05.705ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:53651
35482024-06-21T21:43:05.749ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 43837 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:53651 req_id = c8e1bc81-e739-4547-9b2e-06857bf2135c response_code = 204 uri = /switch-ports
35492024-06-21T21:43:05.749ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:39750
35502024-06-21T21:43:05.752ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3238 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:53651 req_id = cee427eb-1167-45ea-8067-8ecdbe414cc9 response_code = 200 uri = /network-bootstore-config
35512024-06-21T21:43:05.791ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 42199 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:39750 req_id = 33795f0b-50f4-4303-a749-4a2d71b71f32 response_code = 204 uri = /switch-ports
35522024-06-21T21:43:05.791ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 139 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:39750 req_id = 2d64c559-1265-4db8-8062-4b1c67cbf99e response_code = 200 uri = /network-bootstore-config
35532024-06-21T21:43:09.634ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:45536
35542024-06-21T21:43:09.634ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 117 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:45536 req_id = b7d5151c-3a42-4947-b736-30e7bee32d66 response_code = 200 uri = /baseboard
35552024-06-21T21:43:35.461ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60546
35562024-06-21T21:43:35.462ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 246 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60546 req_id = 721c8e72-d1bb-4f76-8983-7d9b24f7d1fd response_code = 200 uri = /v2p
35572024-06-21T21:43:35.566ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:57722
35582024-06-21T21:43:35.617ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 50097 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:57722 req_id = 341ef96e-13aa-46d8-bb22-6dff8a85eb83 response_code = 204 uri = /switch-ports
35592024-06-21T21:43:35.618ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1134 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:57722 req_id = 1fc4ccf0-d305-4acd-abfd-ebcc1821c0f3 response_code = 200 uri = /network-bootstore-config
35602024-06-21T21:43:35.690ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44702
35612024-06-21T21:43:35.691ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 966 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44702 req_id = 33b415dd-b48e-4f80-93b5-233d66088030 response_code = 200 uri = /v2p
35622024-06-21T21:43:35.757ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34460
35632024-06-21T21:43:35.758ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 915 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34460 req_id = 0a58adfb-092b-47e6-8ff8-93b88045bdbd response_code = 200 uri = /v2p
35642024-06-21T21:43:35.834ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:41564
35652024-06-21T21:43:35.878ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44578 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:41564 req_id = d5adc60d-e329-48aa-86fe-ddae5d14c343 response_code = 204 uri = /switch-ports
35662024-06-21T21:43:35.879ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 592 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:41564 req_id = 1bc576d5-6ed0-49d1-961d-b94b039a2068 response_code = 200 uri = /network-bootstore-config
35672024-06-21T21:43:35.919ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:39243
35682024-06-21T21:43:35.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44709 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:39243 req_id = 9af64936-66e8-42de-ba68-6d9b801c27a2 response_code = 204 uri = /switch-ports
35692024-06-21T21:43:35.964ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 95 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:39243 req_id = 8f23500b-c066-4c02-8247-fba0ceb533c2 response_code = 200 uri = /network-bootstore-config
35702024-06-21T21:43:39.668ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:59909
35712024-06-21T21:43:39.668ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:59909 req_id = fdfb3a50-3753-4ff1-b09d-74a826de4a65 response_code = 200 uri = /baseboard
35722024-06-21T21:44:05.294ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:63051
35732024-06-21T21:44:05.294ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:63051 req_id = 499b48bb-70fd-4860-9783-af93115573b9 response_code = 200 uri = /v2p
35742024-06-21T21:44:05.430ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46342
35752024-06-21T21:44:05.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48052 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:46342 req_id = 37438be1-9070-4009-87b0-e498b668644f response_code = 204 uri = /switch-ports
35762024-06-21T21:44:05.479ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46342 req_id = 15f9db7b-8fd2-4cb4-b9e8-0593f7379544 response_code = 200 uri = /network-bootstore-config
35772024-06-21T21:44:05.525ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:33310
35782024-06-21T21:44:05.525ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:33310 req_id = 93cacaef-cac1-4a91-98af-ae3785993595 response_code = 200 uri = /v2p
35792024-06-21T21:44:05.641ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50316
35802024-06-21T21:44:05.641ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 124 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50316 req_id = 7dbf9b37-bb45-4835-a32c-4b535821abd2 response_code = 200 uri = /v2p
35812024-06-21T21:44:05.647ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52307
35822024-06-21T21:44:05.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44587 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52307 req_id = 93ef12d8-0576-44fd-8695-64c95e8225cb response_code = 204 uri = /switch-ports
35832024-06-21T21:44:05.692ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 109 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52307 req_id = 1aab933d-9812-4115-a62b-84d4df10c620 response_code = 200 uri = /network-bootstore-config
35842024-06-21T21:44:05.715ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:43223
35852024-06-21T21:44:05.759ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44118 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:43223 req_id = bb679923-e264-4389-9543-79639d467001 response_code = 204 uri = /switch-ports
35862024-06-21T21:44:05.759ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 112 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:43223 req_id = 50b3b623-b1c8-44bf-af04-c9015452055e response_code = 200 uri = /network-bootstore-config
35872024-06-21T21:44:09.700ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:34560
35882024-06-21T21:44:09.701ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 713 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:34560 req_id = b88977c6-71e0-4e96-a38b-8b38e46a8d78 response_code = 200 uri = /baseboard
35892024-06-21T21:44:35.341ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:36820
35902024-06-21T21:44:35.342ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 841 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:36820 req_id = e7cbf996-7596-4bbb-a37e-ea42cd50e319 response_code = 200 uri = /v2p
35912024-06-21T21:44:35.583ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:35284
35922024-06-21T21:44:35.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 53127 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:35284 req_id = 566664b9-904f-45ff-9c45-af5b7dc96908 response_code = 204 uri = /switch-ports
35932024-06-21T21:44:35.637ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:48074
35942024-06-21T21:44:35.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 198 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:48074 req_id = ad5d83b9-b87c-4a5e-97ef-73f5f17be065 response_code = 200 uri = /v2p
35952024-06-21T21:44:35.637ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 231 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:35284 req_id = 12c0fe31-da26-454d-9b4e-a18c2e201773 response_code = 200 uri = /network-bootstore-config
35962024-06-21T21:44:35.717ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48237
35972024-06-21T21:44:35.718ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48237 req_id = 2dd4d736-0526-4171-8b4c-e2cb866fcf2c response_code = 200 uri = /v2p
35982024-06-21T21:44:35.829ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:48375
35992024-06-21T21:44:35.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45857 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:48375 req_id = 72120775-88ab-4df7-9f1d-e802fcb65f4e response_code = 204 uri = /switch-ports
36002024-06-21T21:44:35.875ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:48375 req_id = 440fa526-af23-41a1-a515-296bb1224d02 response_code = 200 uri = /network-bootstore-config
36012024-06-21T21:44:35.929ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52421
36022024-06-21T21:44:35.975ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46027 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:52421 req_id = e902623a-8dbd-44f9-ae60-e0ef1a781481 response_code = 204 uri = /switch-ports
36032024-06-21T21:44:35.975ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52421 req_id = a09b01e1-386e-4c03-8328-816184740c57 response_code = 200 uri = /network-bootstore-config
36042024-06-21T21:44:39.734ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:33698
36052024-06-21T21:44:39.734ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:33698 req_id = 3c0cdaaf-c5a3-4524-b549-49cfcf8b266a response_code = 200 uri = /baseboard
36062024-06-21T21:44:41.123ZINFOSledAgent (dropshot): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:44452 remote_addr = [fd00:1122:3344:101::d]:37031 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f
36072024-06-21T21:44:41.135ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 12357 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 9d5f2b23-e9d9-4032-b0af-f49b508231e8 response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
36082024-06-21T21:45:05.278ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44557
36092024-06-21T21:45:05.279ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 196 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44557 req_id = 7c1a6137-c84c-426a-a3bc-312257d8e658 response_code = 200 uri = /v2p
36102024-06-21T21:45:05.482ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:34218
36112024-06-21T21:45:05.526ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44746 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:34218 req_id = 6b979a22-a99b-472b-8b18-38bb8fdc3088 response_code = 204 uri = /switch-ports
36122024-06-21T21:45:05.527ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:44308
36132024-06-21T21:45:05.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 995 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:44308 req_id = c4ee328a-e904-4911-8b24-b3af280ed5fe response_code = 200 uri = /v2p
36142024-06-21T21:45:05.528ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 954 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:34218 req_id = 9e3cdf34-8e22-4b41-a787-6456291119fe response_code = 200 uri = /network-bootstore-config
36152024-06-21T21:45:05.606ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63346
36162024-06-21T21:45:05.606ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 174 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63346 req_id = ac57cad7-4f10-47e0-9d24-3ecb95779484 response_code = 200 uri = /v2p
36172024-06-21T21:45:05.688ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:56580
36182024-06-21T21:45:05.732ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44211 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:56580 req_id = 5d9952de-e843-46c7-b93c-18a107af7f95 response_code = 204 uri = /switch-ports
36192024-06-21T21:45:05.732ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:50479
36202024-06-21T21:45:05.736ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3267 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:56580 req_id = 014a6bd0-4001-4941-9b20-8b708607989e response_code = 200 uri = /network-bootstore-config
36212024-06-21T21:45:05.778ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45881 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:50479 req_id = 840ab970-1c41-42f4-a267-688edfd31981 response_code = 204 uri = /switch-ports
36222024-06-21T21:45:05.779ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 751 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:50479 req_id = d13e337a-00a8-4603-b3b0-03fbaeebc816 response_code = 200 uri = /network-bootstore-config
36232024-06-21T21:45:09.770ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:36811
36242024-06-21T21:45:09.770ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:36811 req_id = 4dcc4d31-eb9b-4617-9017-f0bc933df1c5 response_code = 200 uri = /baseboard
36252024-06-21T21:45:11.123ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 813 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 819a72ae-6ee3-422f-b455-57c33ead6d97 response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
36262024-06-21T21:45:35.362ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:41024
36272024-06-21T21:45:35.362ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 164 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:41024 req_id = cb6a0603-d597-40e9-924a-d610f4e5b4a5 response_code = 200 uri = /v2p
36282024-06-21T21:45:35.575ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:42877
36292024-06-21T21:45:35.576ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 977 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:42877 req_id = 59d8ffec-6510-407f-8cbd-7fff41909f9e response_code = 200 uri = /v2p
36302024-06-21T21:45:35.649ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65117
36312024-06-21T21:45:35.695ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46119 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:65117 req_id = c1396135-9778-4da9-95d4-ea066e055b19 response_code = 204 uri = /switch-ports
36322024-06-21T21:45:35.696ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:65117 req_id = 0c0fb908-e3e3-4394-9ed7-d33662187b0d response_code = 200 uri = /network-bootstore-config
36332024-06-21T21:45:35.747ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:61601
36342024-06-21T21:45:35.747ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 160 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:61601 req_id = fcf86cd7-c1c6-4ba6-8d96-6dec82f6cd19 response_code = 200 uri = /v2p
36352024-06-21T21:45:35.848ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:47880
36362024-06-21T21:45:35.895ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46817 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:47880 req_id = 1c63c599-1376-4591-9871-0ba422a75055 response_code = 204 uri = /switch-ports
36372024-06-21T21:45:35.895ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36632
36382024-06-21T21:45:35.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48035 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:36632 req_id = c0b5df65-38a4-49e4-af0e-75111834dd27 response_code = 204 uri = /switch-ports
36392024-06-21T21:45:35.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 117 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:47880 req_id = 6e220ca2-9e12-4704-9d97-ae4d1f5ca165 response_code = 200 uri = /network-bootstore-config
36402024-06-21T21:45:35.943ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 123 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36632 req_id = 62724021-0d8c-4c21-b3f4-4b9c43851a1a response_code = 200 uri = /network-bootstore-config
36412024-06-21T21:45:39.802ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:65185
36422024-06-21T21:45:39.802ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 97 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:65185 req_id = e2af2d89-af3a-40f0-ae77-2d77bec11a2d response_code = 200 uri = /baseboard
36432024-06-21T21:45:41.123ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 496 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 19b81599-524c-4382-aeb8-f95d8faf2073 response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
36442024-06-21T21:46:05.344ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:53595
36452024-06-21T21:46:05.344ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 184 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:53595 req_id = f8a37a7d-4bd5-4975-9ebd-ec89572a5519 response_code = 200 uri = /v2p
36462024-06-21T21:46:05.426ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47417
36472024-06-21T21:46:05.478ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 51778 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:47417 req_id = 05e54e99-6097-44af-9316-1291d6fef0f0 response_code = 204 uri = /switch-ports
36482024-06-21T21:46:05.479ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 115 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47417 req_id = 271ad886-17f3-4793-b451-ba65e0d58b74 response_code = 200 uri = /network-bootstore-config
36492024-06-21T21:46:05.498ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:63411
36502024-06-21T21:46:05.499ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 182 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:63411 req_id = d68c1cac-dc5d-4d5f-ac61-b886ab51440a response_code = 200 uri = /v2p
36512024-06-21T21:46:05.512ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45931
36522024-06-21T21:46:05.512ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 81 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45931 req_id = 235375d5-6785-4af2-90d8-702c2b34b2a3 response_code = 200 uri = /v2p
36532024-06-21T21:46:05.615ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:60048
36542024-06-21T21:46:05.661ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46089 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:60048 req_id = 914f9dc9-c0a0-47b4-be9f-7ff82f81edcb response_code = 204 uri = /switch-ports
36552024-06-21T21:46:05.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:59789
36562024-06-21T21:46:05.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 39615 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:59789 req_id = b3fdd0c5-e3e3-4e98-a451-25de92807c7a response_code = 204 uri = /switch-ports
36572024-06-21T21:46:05.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 126 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:60048 req_id = dc24d12f-3356-4a5e-a706-171f91faa889 response_code = 200 uri = /network-bootstore-config
36582024-06-21T21:46:05.701ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 187 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:59789 req_id = 4f2fb7be-3c99-448f-90a3-05acafae007e response_code = 200 uri = /network-bootstore-config
36592024-06-21T21:46:09.832ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:46661
36602024-06-21T21:46:09.833ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 116 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:46661 req_id = 7e6bd850-6e58-4818-a2dc-be00c40b6a2b response_code = 200 uri = /baseboard
36612024-06-21T21:46:11.124ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1701 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 6ac3e792-95c2-45e2-a69d-8d131deac61d response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
36622024-06-21T21:46:35.531ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:47337
36632024-06-21T21:46:35.532ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 191 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:47337 req_id = 96a2e4b3-a518-424a-93da-84e22e381ded response_code = 200 uri = /v2p
36642024-06-21T21:46:35.611ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61540
36652024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
36662024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36672024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36682024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36692024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
36702024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
36712024-06-21T21:46:35.611ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
36722024-06-21T21:46:35.611ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 395 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:61540 req_id = 7409ba77-81f8-4122-aa12-6f855726c820 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
36732024-06-21T21:46:35.629ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:49322
36742024-06-21T21:46:35.630ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 103 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:49322 req_id = bad2fbe5-2589-4554-a341-18b14872455f response_code = 200 uri = /v2p
36752024-06-21T21:46:35.639ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:42828
36762024-06-21T21:46:35.640ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1008 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:42828 req_id = fea44d73-6821-4167-96e3-c247567662fe response_code = 200 uri = /v2p
36772024-06-21T21:46:35.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:51370
36782024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
36792024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36802024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36812024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36822024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
36832024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
36842024-06-21T21:46:35.711ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
36852024-06-21T21:46:35.711ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 317 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::c]:51370 req_id = 64a28ac7-3b71-40f6-b8dd-37e62a4de336 response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
36862024-06-21T21:46:35.723ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:60231
36872024-06-21T21:46:35.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 43178 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:60231 req_id = 2207c562-9bbe-4f2f-a4f5-38b96ead820d response_code = 204 uri = /switch-ports
36882024-06-21T21:46:35.766ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 94 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:60231 req_id = bb6898de-7ba7-4e6b-85cd-4a9ca9ae1244 response_code = 200 uri = /network-bootstore-config
36892024-06-21T21:46:35.813ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:37036
36902024-06-21T21:46:35.859ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46163 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:37036 req_id = 2c458e92-45e2-49e1-b6e1-3b69ad346b54 response_code = 204 uri = /switch-ports
36912024-06-21T21:46:35.860ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 349 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:37036 req_id = 8420c8b9-e4df-41d3-bd63-024186be86be response_code = 200 uri = /network-bootstore-config
36922024-06-21T21:46:35.938ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:65494
36932024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Ensuring VPC firewall rules file = illumos-utils/src/opte/port_manager.rs:524 rules = [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 87329b95-0651-44f2-adda-4128107eaee1, kind: Service { id: 1acd5e4e-4ad7-4431-b537-a28da2f217bf }, name: Name("external-dns-1acd5e4e-4ad7-4431-b537-a28da2f217bf"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 153, 223])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: e1d06c56-01a0-4b8c-93bb-5d815c89cfbd, kind: Service { id: 8118d247-c8a4-47b6-be18-05eb663745b6 }, name: Name("external-dns-8118d247-c8a4-47b6-be18-05eb663745b6"), ip: 172.30.1.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 241, 8])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(53), last: L4Port(53) }]), filter_protocols: Some([Udp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: 06f950ab-4c74-4cfe-bddf-56696ea9fda9, kind: Service { id: 62aaea40-4286-44d3-b143-827d830cadd5 }, name: Name("nexus-62aaea40-4286-44d3-b143-827d830cadd5"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 159, 118])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 29684b4f-8eb8-47d3-88fd-6ea23f377491, kind: Service { id: f3a19a9d-3117-4bf5-aa39-824f48040fae }, name: Name("nexus-f3a19a9d-3117-4bf5-aa39-824f48040fae"), ip: 172.30.2.5, mac: MacAddr(MacAddr6([168, 64, 37, 255, 193, 73])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }, NetworkInterface { id: 5499fbd2-1f8b-4448-96bd-4f5f1553bf65, kind: Service { id: 3bdb0a0f-906b-4629-8c23-bd2bba1cce07 }, name: Name("nexus-3bdb0a0f-906b-4629-8c23-bd2bba1cce07"), ip: 172.30.2.7, mac: MacAddr(MacAddr6([168, 64, 37, 255, 255, 165])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(80), last: L4Port(80) }, L4PortRange { first: L4Port(443), last: L4Port(443) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }] vni = Vni(100)
36942024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte2" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36952024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte3" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36962024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte1" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(TCP), ports: PortList([80, 443]) }, action: Allow, priority: 65534 }]
36972024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte4" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
36982024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte0" rules = []
36992024-06-21T21:46:35.939ZINFOSledAgent (PortManager): Setting OPTE firewall rules file = illumos-utils/src/opte/port_manager.rs:543 port = "opte5" rules = [FirewallRule { direction: In, filters: Filters { hosts: Any, protocol: Proto(UDP), ports: PortList([53]) }, action: Allow, priority: 65534 }]
37002024-06-21T21:46:35.939ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 339 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:65494 req_id = fd5dafd2-cf93-46c4-9c78-7c3848d86fce response_code = 204 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
37012024-06-21T21:46:36.023ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56147
37022024-06-21T21:46:36.067ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44812 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:56147 req_id = b145a18f-3030-4054-9680-3be0537bd571 response_code = 204 uri = /switch-ports
37032024-06-21T21:46:36.068ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 113 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56147 req_id = 2aab7217-495b-46ab-9e1b-d1d42f9eb037 response_code = 200 uri = /network-bootstore-config
37042024-06-21T21:46:39.865ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:37729
37052024-06-21T21:46:39.865ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:37729 req_id = 2b5551b3-8dd2-4ffb-b52e-6ea937e56638 response_code = 200 uri = /baseboard
37062024-06-21T21:46:41.135ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 632 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 75ad300d-eb06-4db1-bed4-2549017c785b response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
37072024-06-21T21:47:05.255ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:61532
37082024-06-21T21:47:05.256ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 932 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:61532 req_id = 3b1b94b3-2a03-4a57-a381-9c28373336cb response_code = 200 uri = /v2p
37092024-06-21T21:47:05.377ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52698
37102024-06-21T21:47:05.434ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 57093 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52698 req_id = a962aa3e-cecd-4005-b6ea-df4150a879c4 response_code = 204 uri = /switch-ports
37112024-06-21T21:47:05.435ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52698 req_id = 7cbd4fb7-1d15-4abb-b76a-0d90c54c749d response_code = 200 uri = /network-bootstore-config
37122024-06-21T21:47:05.527ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:64700
37132024-06-21T21:47:05.527ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 157 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:64700 req_id = ca15a862-ae98-462c-97bc-d51496efc80d response_code = 200 uri = /v2p
37142024-06-21T21:47:05.534ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:56456
37152024-06-21T21:47:05.534ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 71 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:56456 req_id = 6883f86f-41bd-458e-9dd5-2fb63b6a8a80 response_code = 200 uri = /v2p
37162024-06-21T21:47:05.640ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34536
37172024-06-21T21:47:05.686ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46209 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:34536 req_id = 206f1bee-7d9b-4139-ac6c-816c296e0c86 response_code = 204 uri = /switch-ports
37182024-06-21T21:47:05.686ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:45601
37192024-06-21T21:47:05.689ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3285 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34536 req_id = 05247109-e657-495e-9714-684813b5abfb response_code = 200 uri = /network-bootstore-config
37202024-06-21T21:47:05.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 44953 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:45601 req_id = 2f63e0b2-5288-4fd4-8ffa-635a1597d9dc response_code = 204 uri = /switch-ports
37212024-06-21T21:47:05.731ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 147 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:45601 req_id = 8d6dee1d-96da-4df1-b1eb-931274953aac response_code = 200 uri = /network-bootstore-config
37222024-06-21T21:47:09.897ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:64196
37232024-06-21T21:47:09.897ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 93 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:64196 req_id = 5c78158a-ca78-4c62-911f-f98b116c207e response_code = 200 uri = /baseboard
37242024-06-21T21:47:11.183ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1217 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 29ca673f-cd45-45c7-9878-a0880c613a2b response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
37252024-06-21T21:47:13.399ZINFOSledAgent (auto-cleanup-task): running automatic periodic zone bundle cleanup file = sled-agent/src/zone_bundle.rs:300
37262024-06-21T21:47:13.399ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
37272024-06-21T21:47:13.437ZINFOSledAgent (auto-cleanup-task): top of bundle cleanup loop file = sled-agent/src/zone_bundle.rs:289 next_cleanup = Instant { tv_sec: 5795, tv_nsec: 27893541 } time_to_next_cleanup = 599.999995549s
37282024-06-21T21:47:26.972ZINFOSledAgent (DumpSetup-worker): No archival destination for crash dumps yet chosen. file = sled-agent/src/dump_setup.rs:831
37292024-06-21T21:47:35.568ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:44578
37302024-06-21T21:47:35.570ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1380 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:44578 req_id = 9ce555f2-67e5-40ce-b71c-fd6c8623992e response_code = 200 uri = /v2p
37312024-06-21T21:47:35.635ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:52831
37322024-06-21T21:47:35.635ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 111 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:52831 req_id = f7e23dc0-65a7-4231-aa5a-6a59c8eb5cc9 response_code = 200 uri = /v2p
37332024-06-21T21:47:35.661ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:58137
37342024-06-21T21:47:35.708ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47619 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:58137 req_id = f712414a-4e5d-4feb-878d-cba1232c96f0 response_code = 204 uri = /switch-ports
37352024-06-21T21:47:35.710ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1031 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:58137 req_id = 6360c9b3-e463-4288-9caa-45c7b835e7d8 response_code = 200 uri = /network-bootstore-config
37362024-06-21T21:47:35.811ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:52588
37372024-06-21T21:47:35.811ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 218 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:52588 req_id = 66e14f26-7061-4487-ae45-8049d6783ac4 response_code = 200 uri = /v2p
37382024-06-21T21:47:35.940ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:55657
37392024-06-21T21:47:35.989ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 48755 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:55657 req_id = 8c02c7a7-5e07-4aa0-8c71-f36d0a6d49a1 response_code = 204 uri = /switch-ports
37402024-06-21T21:47:35.989ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:49031
37412024-06-21T21:47:35.992ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 3409 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:55657 req_id = 2f3d13ae-924b-4f02-8167-ef852786be09 response_code = 200 uri = /network-bootstore-config
37422024-06-21T21:47:36.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 46665 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:49031 req_id = 0fa0f356-69bb-41bc-bb91-e56c69679368 response_code = 204 uri = /switch-ports
37432024-06-21T21:47:36.036ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 166 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:49031 req_id = f5a4df2c-e603-47d3-bde0-b4379f77ff91 response_code = 200 uri = /network-bootstore-config
37442024-06-21T21:47:39.930ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:38144
37452024-06-21T21:47:39.930ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 59 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:38144 req_id = a993ecf1-07a9-4045-a4ce-59c18471d743 response_code = 200 uri = /baseboard
37462024-06-21T21:47:41.182ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 537 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 4d830699-e16f-4ac7-9555-f37ba1df2f64 response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
37472024-06-21T21:47:56.338ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:62446
37482024-06-21T21:47:56.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 827 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:62446 req_id = 0db9a1da-622f-4741-a19f-4b8493269b02 response_code = 200 uri = /v2p
37492024-06-21T21:47:56.339ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:631 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11503171) }
37502024-06-21T21:47:56.339ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 106 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:62446 req_id = 66789f1a-3611-47fa-a633-28be5c71dc31 response_code = 204 uri = /v2p
37512024-06-21T21:47:56.517ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:55649
37522024-06-21T21:47:56.530ZINFOSledAgent (InstanceManager): ensuring instance is registered file = sled-agent/src/instance_manager.rs:548 hardware = InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(11503171))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-63db0df550c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 0a1f9dd3-0ab2-4c85-9591-e2355a975ba8, read_only_parent: Some(Volume { block_size: 512, id: 67ec9101-4b43-4093-a2e9-70b0ef4eb529, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: da4c6c6a-aa11-4c27-bb09-841cb9533bd1, key: Some("VsR3vyTAY+UXrYzJE9u/pr9TpkYPTwVKlza2Bnd3k3w="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::16]:19002", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0a1f9dd3-0ab2-4c85-9591-e2355a975ba8, key: Some("f+RBp+MwWUlHxGnnkjNnadh7+cwPieq3z1R17M72Gq0="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::19]:19000", "[fd00:1122:3344:101::15]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAw/K3VWNVYAAD8rdVYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAMPyt1VjVWAAA/K3VWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJmZmIxMzJkYy1hZDAxLTRhMjgtYjgxZC01OTI4NGMxZDM2OTgiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTHFiVWdPanVhYWFUS0p0WURYbG55STQ2Q0NHZFNvV3RQR1RyQ2lrZjdJZiJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") } instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 instance_runtime = InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z } metadata = InstanceMetadata { silo_id: d34f0978-28ba-47c2-ba5e-da232e57f4de, project_id: 02826343-f07c-48e9-b8b7-43743c36bd41 } propolis_addr = [fd00:1122:3344:101::1:0]:12400 propolis_id = 0ced2394-b8aa-45d3-9d56-9a5d4819298f vmm_runtime = VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-06-21T21:47:56.075605Z }
37532024-06-21T21:47:56.531ZINFOSledAgent (InstanceManager): registering new instance file = sled-agent/src/instance_manager.rs:582 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 (instance)
37542024-06-21T21:47:56.532ZINFOSledAgent (InstanceManager): initializing new Instance file = sled-agent/src/instance.rs:942 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 propolis_id = 0ced2394-b8aa-45d3-9d56-9a5d4819298f state = InstanceInitialState { hardware: InstanceHardware { properties: InstanceProperties { ncpus: InstanceCpuCount(2), memory: ByteCount(1073741824), hostname: Hostname("localshark") }, nics: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], source_nat: SourceNatConfig { ip: 10.151.2.175, first_port: 0, last_port: 16383 }, ephemeral_ip: Some(10.151.2.176), floating_ips: [], firewall_rules: [VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], filter_hosts: None, filter_ports: None, filter_protocols: Some([Icmp]), action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], filter_hosts: Some([Vpc(Vni(11503171))]), filter_ports: None, filter_protocols: None, action: Allow, priority: VpcFirewallRulePriority(65534) }, VpcFirewallRule { status: Enabled, direction: Inbound, targets: [NetworkInterface { id: b9889eb0-e1af-4305-b861-a14299ad9c84, kind: Instance { id: ffb132dc-ad01-4a28-b81d-59284c1d3698 }, name: Name("net0"), ip: 172.30.0.5, mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), subnet: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), vni: Vni(11503171), primary: true, slot: 0 }], filter_hosts: None, filter_ports: Some([L4PortRange { first: L4Port(22), last: L4Port(22) }]), filter_protocols: Some([Tcp]), action: Allow, priority: VpcFirewallRulePriority(65534) }], dhcp_config: DhcpConfig { dns_servers: [1.1.1.1, 9.9.9.9], host_domain: None, search_domains: [] }, disks: [DiskRequest { device: "nvme", name: "disk-63db0df550c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 0a1f9dd3-0ab2-4c85-9591-e2355a975ba8, read_only_parent: Some(Volume { block_size: 512, id: 67ec9101-4b43-4093-a2e9-70b0ef4eb529, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: da4c6c6a-aa11-4c27-bb09-841cb9533bd1, key: Some("VsR3vyTAY+UXrYzJE9u/pr9TpkYPTwVKlza2Bnd3k3w="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::16]:19002", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0a1f9dd3-0ab2-4c85-9591-e2355a975ba8, key: Some("f+RBp+MwWUlHxGnnkjNnadh7+cwPieq3z1R17M72Gq0="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::19]:19000", "[fd00:1122:3344:101::15]:19000"] } }] } }], cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAw/K3VWNVYAAD8rdVYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAMPyt1VjVWAAA/K3VWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJmZmIxMzJkYy1hZDAxLTRhMjgtYjgxZC01OTI4NGMxZDM2OTgiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTHFiVWdPanVhYWFUS0p0WURYbG55STQ2Q0NHZFNvV3RQR1RyQ2lrZjdJZiJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA") }, instance_runtime: InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z }, vmm_runtime: VmmRuntimeState { state: Starting, gen: Generation(1), time_updated: 2024-06-21T21:47:56.075605Z }, propolis_addr: [fd00:1122:3344:101::1:0]:12400 }
37552024-06-21T21:47:56.535ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 17391 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:55649 req_id = f7a59257-18e3-43a3-a02d-e2588b3405bb response_code = 200 uri = /instances/ffb132dc-ad01-4a28-b81d-59284c1d3698
37562024-06-21T21:47:56.673ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:65458
37572024-06-21T21:47:56.674ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 1556 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:65458 req_id = 4ba39eb8-d38d-4c06-ba35-565836d82101 response_code = 200 uri = /instances/ffb132dc-ad01-4a28-b81d-59284c1d3698/state
37582024-06-21T21:47:56.696ZINFOSledAgent (PortManager): Created OPTE port file = illumos-utils/src/opte/port_manager.rs:393 port = Port { inner: PortInner { name: "opte6", ip: 172.30.0.5, mac: MacAddr6([168, 64, 37, 246, 148, 104]), slot: 0, vni: Vni { inner: 11503171 }, gateway: Gateway { mac: MacAddr6([168, 64, 37, 255, 119, 119]), ip: 172.30.0.1 }, vnic: "vopte6" } }
37592024-06-21T21:47:56.696ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
37602024-06-21T21:47:56.730ZINFOSledAgent (InstanceManager): Configuring new Omicron zone: oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f file = illumos-utils/src/zone.rs:315 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37612024-06-21T21:47:56.751ZINFOSledAgent (InstanceManager): Installing Omicron zone: oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f file = illumos-utils/src/zone.rs:354 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37622024-06-21T21:47:58.166ZINFOSledAgent (InstanceManager): Profile for oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f: <!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"> <service_bundle type="profile" name="omicron"> <service version="1" type="service" name="system/illumos/propolis-server"> <instance enabled="true" name="default"> <property_group type="application" name="config"> <propval type="astring" name="datalink" value="oxControlInstance0"/> <propval type="astring" name="gateway" value="fd00:1122:3344:101::1"/> <propval type="astring" name="listen_addr" value="fd00:1122:3344:101::1:0"/> <propval type="astring" name="listen_port" value="12400"/> <propval type="astring" name="metric_addr" value="[fd00:1122:3344:101::a]:12221"/> </property_group> </instance> </service> </service_bundle> file = sled-agent/src/profile.rs:34 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37632024-06-21T21:47:58.166ZINFOSledAgent (InstanceManager): Zone booting file = illumos-utils/src/running_zone.rs:513 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37642024-06-21T21:47:58.344ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 50.916132ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37652024-06-21T21:47:58.405ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 61.493466ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37662024-06-21T21:47:58.482ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 103.864635ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37672024-06-21T21:47:58.619ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 391.038143ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37682024-06-21T21:47:59.023ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 466.755916ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37692024-06-21T21:47:59.500ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 1.463413787s file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37702024-06-21T21:48:00.974ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 929.494962ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37712024-06-21T21:48:01.914ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 759.876956ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37722024-06-21T21:48:02.684ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 566.235264ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 zone = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
37732024-06-21T21:48:03.280ZINFOSledAgent (InstanceManager): Started propolis in zone: oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f file = sled-agent/src/instance.rs:1427 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37742024-06-21T21:48:03.293ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 57.724767ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37752024-06-21T21:48:03.361ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 149.388942ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37762024-06-21T21:48:03.519ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 103.845635ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37772024-06-21T21:48:03.632ZWARNSledAgent (InstanceManager): wait for service Some("oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f") failed: Property not found. retry in 589.523057ms file = illumos-utils/src/svc.rs:36 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37782024-06-21T21:48:04.230ZINFOSledAgent (InstanceManager): Propolis SMF service is online file = sled-agent/src/instance.rs:1436 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37792024-06-21T21:48:04.266ZINFOSledAgent (InstanceManager): Propolis HTTP server online file = sled-agent/src/instance.rs:1450 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37802024-06-21T21:48:04.270ZINFOSledAgent (InstanceManager): Sending ensure request to propolis: InstanceEnsureRequest { cloud_init_bytes: Some("6zyQTVNXSU40LjEAAgEBAAIAAioA+AEAKgBAAAAAAAAAAAAAAAApeFY0EmNpZGF0YSAgICAgRkFUMTIgICAOH75bfKwiwHQLVrQOuwcAzRBe6/Ay5M0WzRnr/lRoaXMgaXMgbm90IGEgYm9vdGFibGUgZGlzay4gIFBsZWFzZSBpbnNlcnQgYSBib290YWJsZSBmbG9wcHkgYW5kDQpwcmVzcyBhbnkga2V5IHRvIHRyeSBhZ2FpbiAuLi4gDQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAVar4////DwAAAAAAAAAA8P//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////D/j///8PAAAAAAAAAADw//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////8PY2lkYXRhICAgICAIAAAAAAAAAAAAAAAAAAAAAAAAAABBbQBlAHQAYQAtAA8A1GQAYQB0AGEAAAD//wAA/////01FVEEtRH4xICAgAAAw/K3VWNVYAAD8rdVYAgC3AAAAQXUAcwBlAHIALQAPALBkAGEAdABhAAAA//8AAP////9VU0VSLUR+MSAgIAAAMPyt1VjVWAAA/K3VWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHsiaW5zdGFuY2UtaWQiOiJmZmIxMzJkYy1hZDAxLTRhMjgtYjgxZC01OTI4NGMxZDM2OTgiLCJsb2NhbC1ob3N0bmFtZSI6ImxvY2Fsc2hhcmsiLCJwdWJsaWMta2V5cyI6WyJzc2gtZWQyNTUxOSBBQUFBQzNOemFDMWxaREkxTlRFNUFBQUFJTHFiVWdPanVhYWFUS0p0WURYbG55STQ2Q0NHZFNvV3RQR1RyQ2lrZjdJZiJdfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), disks: [DiskRequest { device: "nvme", name: "disk-63db0df550c", read_only: false, slot: Slot(0), volume_construction_request: Volume { block_size: 512, id: 0a1f9dd3-0ab2-4c85-9591-e2355a975ba8, read_only_parent: Some(Volume { block_size: 512, id: 67ec9101-4b43-4093-a2e9-70b0ef4eb529, read_only_parent: None, sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 2, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: da4c6c6a-aa11-4c27-bb09-841cb9533bd1, key: Some("VsR3vyTAY+UXrYzJE9u/pr9TpkYPTwVKlza2Bnd3k3w="), key_pem: None, lossy: false, read_only: true, root_cert_pem: None, target: ["[fd00:1122:3344:101::13]:19001", "[fd00:1122:3344:101::16]:19002", "[fd00:1122:3344:101::18]:19001"] } }] }), sub_volumes: [Region { block_size: 512, blocks_per_extent: 131072, extent_count: 32, gen: 1, opts: CrucibleOpts { cert_pem: None, control: None, flush_timeout: None, id: 0a1f9dd3-0ab2-4c85-9591-e2355a975ba8, key: Some("f+RBp+MwWUlHxGnnkjNnadh7+cwPieq3z1R17M72Gq0="), key_pem: None, lossy: false, read_only: false, root_cert_pem: None, target: ["[fd00:1122:3344:101::18]:19002", "[fd00:1122:3344:101::19]:19000", "[fd00:1122:3344:101::15]:19000"] } }] } }], migrate: None, nics: [NetworkInterfaceRequest { name: "vopte6", slot: Slot(0) }], properties: InstanceProperties { bootrom_id: 00000000-0000-0000-0000-000000000000, description: "Test description", id: ffb132dc-ad01-4a28-b81d-59284c1d3698, image_id: 00000000-0000-0000-0000-000000000000, memory: 1024, metadata: InstanceMetadata { project_id: 02826343-f07c-48e9-b8b7-43743c36bd41, silo_id: d34f0978-28ba-47c2-ba5e-da232e57f4de }, name: "localshark", vcpus: 2 } } file = sled-agent/src/instance.rs:689 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37812024-06-21T21:48:04.383ZINFOSledAgent (InstanceManager): result of instance_ensure call is Ok(InstanceEnsureResponse { migrate: None }) file = sled-agent/src/instance.rs:691 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37822024-06-21T21:48:04.384ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Starting), migration_status: NoMigration, time: 2024-06-21T21:48:04.384164609Z } file = sled-agent/src/instance.rs:573 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37832024-06-21T21:48:04.384ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:595 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 new_instance_state = InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z } new_vmm_state = VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-06-21T21:48:04.384164609Z } propolis_id = 0ced2394-b8aa-45d3-9d56-9a5d4819298f
37842024-06-21T21:48:04.384ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:492 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z }, propolis_id: 0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis), vmm_state: VmmRuntimeState { state: Starting, gen: Generation(2), time_updated: 2024-06-21T21:48:04.384164609Z }, migration_state: None }
37852024-06-21T21:48:04.471ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Running), migration_status: NoMigration, time: 2024-06-21T21:48:04.471204316Z } file = sled-agent/src/instance.rs:573 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
37862024-06-21T21:48:04.471ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:595 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 new_instance_state = InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z } new_vmm_state = VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-06-21T21:48:04.471204316Z } propolis_id = 0ced2394-b8aa-45d3-9d56-9a5d4819298f
37872024-06-21T21:48:04.471ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:492 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z }, propolis_id: 0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis), vmm_state: VmmRuntimeState { state: Running, gen: Generation(3), time_updated: 2024-06-21T21:48:04.471204316Z }, migration_state: None }
37882024-06-21T21:48:05.287ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:45713
37892024-06-21T21:48:05.288ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 649 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:45713 req_id = b8359114-4d38-4338-a404-ccb8047ea11f response_code = 200 uri = /instances/ffb132dc-ad01-4a28-b81d-59284c1d3698/state
37902024-06-21T21:48:05.309ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:49522
37912024-06-21T21:48:05.309ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 201 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:49522 req_id = e22e3fa9-a413-4955-9555-29e87d19f765 response_code = 200 uri = /v2p
37922024-06-21T21:48:05.518ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:60932
37932024-06-21T21:48:05.518ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 125 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:60932 req_id = 611b4d9b-af3a-40db-ac66-f6fff53ace65 response_code = 200 uri = /instances/ffb132dc-ad01-4a28-b81d-59284c1d3698/state
37942024-06-21T21:48:05.558ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:36416
37952024-06-21T21:48:05.558ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 85 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:36416 req_id = b07ba77d-c940-44e4-ba65-066ca64fd4a8 response_code = 200 uri = /instances/ffb132dc-ad01-4a28-b81d-59284c1d3698/state
37962024-06-21T21:48:05.609ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52603
37972024-06-21T21:48:05.659ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 50412 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::a]:52603 req_id = 2a98abbd-6d1c-467c-b12e-2832a186a606 response_code = 204 uri = /switch-ports
37982024-06-21T21:48:05.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38706
37992024-06-21T21:48:05.659ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:34269
38002024-06-21T21:48:05.660ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 208 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:52603 req_id = bc1dcc19-7d61-47bd-aee5-3dec6c3bb13e response_code = 200 uri = /network-bootstore-config
38012024-06-21T21:48:05.660ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 287 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38706 req_id = 056bc725-376b-4e64-bc7e-9d35c0c32580 response_code = 200 uri = /v2p
38022024-06-21T21:48:05.660ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 299 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:34269 req_id = 23e1bbe1-b855-4e3a-97c8-4d99be545cf0 response_code = 200 uri = /v2p
38032024-06-21T21:48:05.729ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:54111
38042024-06-21T21:48:05.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 45440 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::b]:54111 req_id = b8b8cd01-cb93-4895-9735-0c6be0fd0382 response_code = 204 uri = /switch-ports
38052024-06-21T21:48:05.775ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 121 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:54111 req_id = eacf141e-aa7d-466f-9c38-90481007bcdd response_code = 200 uri = /network-bootstore-config
38062024-06-21T21:48:05.780ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::c]:57593
38072024-06-21T21:48:05.828ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 47909 local_addr = [fd00:1122:3344:101::1]:12345 method = POST remote_addr = [fd00:1122:3344:101::c]:57593 req_id = 7bf95355-e7cc-48b2-ab00-46db42bd98ce response_code = 204 uri = /switch-ports
38082024-06-21T21:48:05.828ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 153 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::c]:57593 req_id = c14d7039-9471-435a-8921-1d8b73b45684 response_code = 200 uri = /network-bootstore-config
38092024-06-21T21:48:09.963ZINFOSledAgent (dropshot (BootstrapAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 remote_addr = [fdb0:18c0:4d0d:a02a::2]:51190
38102024-06-21T21:48:09.963ZINFOSledAgent (dropshot (BootstrapAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 79 local_addr = [fdb0:18c0:4d0d:a02a::1]:80 method = GET remote_addr = [fdb0:18c0:4d0d:a02a::2]:51190 req_id = 8e8fd481-b779-4dcf-b2ab-a378c4d0ec3d response_code = 200 uri = /baseboard
38112024-06-21T21:48:11.182ZINFOSledAgent (dropshot): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 570 local_addr = [fd00:1122:3344:101::1]:44452 method = GET remote_addr = [fd00:1122:3344:101::d]:37031 req_id = 196b3f7a-63cf-44d7-bd65-2adbb91ea51d response_code = 200 sled_id = a27c09e4-c9a4-4472-b248-8c2ae9bae00f uri = /a27c09e4-c9a4-4472-b248-8c2ae9bae00f
38122024-06-21T21:48:14.168ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:52685
38132024-06-21T21:48:14.169ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 752 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::a]:52685 req_id = e7c33376-4868-4e58-9b63-7b549fd6e259 response_code = 200 uri = /instances/ffb132dc-ad01-4a28-b81d-59284c1d3698/state
38142024-06-21T21:48:14.169ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Stopping), migration_status: NoMigration, time: 2024-06-21T21:48:14.169356828Z } file = sled-agent/src/instance.rs:573 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38152024-06-21T21:48:14.169ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:595 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 new_instance_state = InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z } new_vmm_state = VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-06-21T21:48:14.169356828Z } propolis_id = 0ced2394-b8aa-45d3-9d56-9a5d4819298f
38162024-06-21T21:48:14.169ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:492 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: Some(0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis)), dst_propolis_id: None, migration_id: None, gen: Generation(3), time_updated: 2024-06-21T21:47:56.123691756Z }, propolis_id: 0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis), vmm_state: VmmRuntimeState { state: Stopping, gen: Generation(5), time_updated: 2024-06-21T21:48:14.169356828Z }, migration_state: None }
38172024-06-21T21:48:14.235ZINFOSledAgent (InstanceManager): Observing new propolis state: ObservedPropolisState { vmm_state: PropolisInstanceState(Destroyed), migration_status: NoMigration, time: 2024-06-21T21:48:14.235572691Z } file = sled-agent/src/instance.rs:573 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38182024-06-21T21:48:14.235ZINFOSledAgent (InstanceManager): updated state after observing Propolis state change file = sled-agent/src/instance.rs:595 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 new_instance_state = InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-06-21T21:48:14.235572691Z } new_vmm_state = VmmRuntimeState { state: Destroyed, gen: Generation(6), time_updated: 2024-06-21T21:48:14.235572691Z } propolis_id = 0ced2394-b8aa-45d3-9d56-9a5d4819298f
38192024-06-21T21:48:14.235ZINFOSledAgent (InstanceManager): terminating VMM that has exited file = sled-agent/src/instance.rs:610 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38202024-06-21T21:48:14.235ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
38212024-06-21T21:48:14.235ZINFOSledAgent (StorageManager): Received GetLatestResources(...) file = sled-storage/src/manager.rs:381
38222024-06-21T21:48:14.235ZINFOSledAgent (ZoneBundler): creating zone bundle context = ZoneBundleContext { storage_dirs: ["/pool/int/89c103d3-b501-4e1d-8d78-ad9e833f154c/debug/bundle/zone", "/pool/int/9af8bc90-90a3-45f5-b12b-12fd9904a27e/debug/bundle/zone"], cause: TerminatedInstance, extra_log_dirs: ["/pool/ext/5b452253-5f6d-4617-91b5-3f263edef77d/crypt/debug", "/pool/ext/41b82b1f-5def-467c-bc86-5c87195bd788/crypt/debug", "/pool/ext/44b9c17b-6e37-45fe-b9da-6182ccbe59a4/crypt/debug", "/pool/ext/e6f1c1a1-ab63-45ad-a43f-153910e0494a/crypt/debug", "/pool/ext/8ce4d8fb-4017-46e3-ac55-3c40125a531b/crypt/debug", "/pool/ext/a0c5f5f0-1cc5-45cc-9434-17b4abf3b972/crypt/debug", "/pool/ext/85a7fd4f-88d4-408c-a832-0d457152c72f/crypt/debug", "/pool/ext/6b2e5449-891f-4dfd-8e2d-78189eabe443/crypt/debug", "/pool/ext/3ae04f9e-dd4d-4980-8e86-31c81824c907/crypt/debug"] } file = sled-agent/src/zone_bundle.rs:437 zone_name = oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f
38232024-06-21T21:48:14.851ZINFOSledAgent (ZoneBundler): finished zone bundle file = sled-agent/src/zone_bundle.rs:1230 metadata = ZoneBundleMetadata { id: ZoneBundleId { zone_name: "oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f", bundle_id: 199907a5-85ad-46d9-80f6-96c240eacdfd }, time_created: 2024-06-21T21:48:14.236206364Z, version: 0, cause: TerminatedInstance }
38242024-06-21T21:48:14.851ZWARNSledAgent (InstanceManager): Halting and removing zone: oxz_propolis-server_0ced2394-b8aa-45d3-9d56-9a5d4819298f file = sled-agent/src/instance.rs:784 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38252024-06-21T21:48:16.585ZINFOSledAgent (InstanceManager): halt_and_remove_logged: Previous zone state: Running file = illumos-utils/src/zone.rs:269 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38262024-06-21T21:48:16.612ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:492 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-06-21T21:48:14.235572691Z }, propolis_id: 0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-06-21T21:48:16.612099245Z }, migration_state: None }
38272024-06-21T21:48:16.711ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::b]:38745
38282024-06-21T21:48:16.712ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 229 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::b]:38745 req_id = 4ab36b64-c28c-41e5-a7d6-b185ac6cf1db response_code = 200 uri = /v2p
38292024-06-21T21:48:16.712ZINFOSledAgent (PortManager): Mapping virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:631 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11503171) }
38302024-06-21T21:48:16.712ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 107 local_addr = [fd00:1122:3344:101::1]:12345 method = PUT remote_addr = [fd00:1122:3344:101::b]:38745 req_id = fbc04eb6-bf51-4cdc-9bea-d6aecae457c2 response_code = 204 uri = /v2p
38312024-06-21T21:48:16.909ZINFOSledAgent (InstanceManager): Publishing instance state update to Nexus file = sled-agent/src/instance.rs:492 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698 state = SledInstanceState { instance_state: InstanceRuntimeState { propolis_id: None, dst_propolis_id: None, migration_id: None, gen: Generation(5), time_updated: 2024-06-21T21:48:14.235572691Z }, propolis_id: 0ced2394-b8aa-45d3-9d56-9a5d4819298f (propolis), vmm_state: VmmRuntimeState { state: Destroyed, gen: Generation(7), time_updated: 2024-06-21T21:48:16.612099245Z }, migration_state: None }
38322024-06-21T21:48:16.913ZINFOSledAgent (InstanceManager): State monitoring task complete file = sled-agent/src/instance.rs:727 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38332024-06-21T21:48:16.958ZERROSledAgent (InstanceManager): Failed to publish state to Nexus, will not retry: Notification(Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "0ce803e7-67d3-4636-91b3-4c8b60c97406", "content-length": "177", "date": "Fri, 21 Jun 2024 21:48:16 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: instance with id \"ffb132dc-ad01-4a28-b81d-59284c1d3698\"", request_id: "0ce803e7-67d3-4636-91b3-4c8b60c97406" }) file = sled-agent/src/instance.rs:559 instance_id = ffb132dc-ad01-4a28-b81d-59284c1d3698
38342024-06-21T21:48:17.039ZINFOSledAgent (dropshot (SledAgent)): accepted connection file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:766 local_addr = [fd00:1122:3344:101::1]:12345 remote_addr = [fd00:1122:3344:101::a]:46092
38352024-06-21T21:48:17.039ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 167 local_addr = [fd00:1122:3344:101::1]:12345 method = GET remote_addr = [fd00:1122:3344:101::a]:46092 req_id = eeadcbcd-6ec3-43c3-919c-2c83905c8620 response_code = 200 uri = /v2p
38362024-06-21T21:48:17.040ZINFOSledAgent (PortManager): Clearing mapping of virtual NIC to physical host file = illumos-utils/src/opte/port_manager.rs:671 mapping = VirtualNetworkInterfaceHost { virtual_ip: 172.30.0.5, virtual_mac: MacAddr(MacAddr6([168, 64, 37, 246, 148, 104])), physical_host_ip: fd00:1122:3344:101::1, vni: Vni(11503171) }
38372024-06-21T21:48:17.040ZINFOSledAgent (dropshot (SledAgent)): request completed file = /home/build/.cargo/git/checkouts/dropshot-a4a923d29dccc492/0cd0e82/dropshot/src/server.rs:878 latency_us = 726 local_addr = [fd00:1122:3344:101::1]:12345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:46092 req_id = 19efa58a-4a4f-4416-bebc-c746ae159b47 response_code = 204 uri = /v2p